Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 164 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -446,8 +446,9 @@ Pub/Sub dashboard actions are available under `/dashboard/pubsub`:
| --- | --- | --- |
| PostgreSQL wire endpoint | Yes | Listens on `127.0.0.1:15439` by default for local Redshift-style SQL clients. |
| PostgreSQL simple query protocol | Yes | Covers `psql` smoke workflows. |
| TLS on the SQL endpoint | Yes | Off by default. Set `services.redshift.tls.certFile` and `services.redshift.tls.keyFile` (PEM; certificate chain leaf first, key in PKCS#8, PKCS#1, or SEC1; paths relative to where `devcloud up` runs) and clients that send an SSLRequest get TLS 1.2 or 1.3 with that certificate, so `sslmode=verify-full` works when the client trusts the issuing CA and the certificate names the host it connects to. devcloud does not issue certificates. Plain connections are still accepted unless `services.redshift.tls.require: true`. The standalone binary reads `DEVCLOUD_REDSHIFT_TLS_CERT_FILE`, `DEVCLOUD_REDSHIFT_TLS_KEY_FILE`, and `DEVCLOUD_REDSHIFT_TLS_REQUIRE`. |
| PostgreSQL extended query protocol | Partial | Supports Parse, Bind, Describe, Execute, Sync, Close, text bind parameters, portal resume, and safe unsupported errors for binary formats. |
| PostgreSQL execution backend | Yes | Default backend is managed local PostgreSQL; explicit memory fallback remains available for development continuity. Each SQL client connection gets its own PostgreSQL session, so temp tables, `SET`s, and transactions stay with that client and end when it disconnects. The SQL endpoint does not offer TLS (`sslmode=disable`/`prefer` clients only). |
| PostgreSQL execution backend | Yes | Default backend is managed local PostgreSQL; explicit memory fallback remains available for development continuity. Each SQL client connection gets its own PostgreSQL session, so temp tables, `SET`s, and transactions stay with that client and end when it disconnects. See TLS below. |
| Redshift SQL translation | Partial | Handles local subsets for DDL/DML, Redshift table attributes, CTAS/views/materialized-view metadata, function rewrites, COPY, and UNLOAD. |
| COPY from local S3 | Partial | `COPY <table> FROM 's3://bucket/prefix'` loads every object under the prefix from the local S3 store into the PostgreSQL backend, on the client's own session (temp tables and open transactions work): default text format with `DELIMITER`, `CSV` / `FORMAT AS CSV`, `JSON 'auto'` (or `'auto ignorecase'`), `GZIP`, `IGNOREHEADER`, `NULL AS`, `EMPTYASNULL`, `BLANKSASNULL`; an empty field is NULL except in character columns, as on Redshift. `IAM_ROLE` / `CREDENTIALS` / `REGION` are accepted and ignored. Column lists, `MANIFEST`, JSONPaths, other compressions, and columnar formats are refused. The memory backend keeps its own CSV/JSON COPY. |
| UNLOAD to local S3 | Partial | Memory backend only; the PostgreSQL backend does not run UNLOAD yet. |
Expand Down
18 changes: 18 additions & 0 deletions orchestrator/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,16 @@ pub struct RedshiftServiceConfig {
pub data_api: RedshiftDataApiConfig,
pub sql: RedshiftSqlConfig,
pub copy_unload: RedshiftCopyUnloadConfig,
pub tls: RedshiftTlsConfig,
}

/// TLS for the Redshift SQL endpoint: PEM files the user supplies.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct RedshiftTlsConfig {
pub cert_file: String,
pub key_file: String,
/// Refuse SQL clients that do not negotiate TLS.
pub require: bool,
}

#[derive(Debug, Clone, PartialEq, Eq, Default)]
Expand Down Expand Up @@ -581,6 +591,7 @@ pub fn default_config() -> Config {
enable_local_s3: true,
max_input_row_bytes: 4 * 1024 * 1024,
},
tls: RedshiftTlsConfig::default(),
},
redis: RedisServiceConfig {
enabled: false,
Expand Down Expand Up @@ -1078,6 +1089,13 @@ pub fn apply_config_value(cfg: &mut Config, path: &[String], value: &str) -> io:
"services.redshift.sql.defaultSearchPath" => {
cfg.services.redshift.sql.default_search_path = value.to_string()
}
"services.redshift.tls.certFile" => {
cfg.services.redshift.tls.cert_file = trim_quotes(value)
}
"services.redshift.tls.keyFile" => cfg.services.redshift.tls.key_file = trim_quotes(value),
"services.redshift.tls.require" => {
cfg.services.redshift.tls.require = parse_bool("services.redshift.tls.require", value)?
}
"services.redshift.copyUnload.enableLocalS3" => {
cfg.services.redshift.copy_unload.enable_local_s3 =
parse_bool("services.redshift.copyUnload.enableLocalS3", value)?
Expand Down
6 changes: 6 additions & 0 deletions orchestrator/src/services/redshift.rs
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,11 @@ pub async fn run(
let sql_addr = format!("127.0.0.1:{}", cfg.server.redshift_port);
let api_addr = format!("127.0.0.1:{}", cfg.server.redshift_api_port);

let tls = &cfg.services.redshift.tls;
let sql_tls =
devcloud_redshift::tls::sql_tls_from_files(&tls.cert_file, &tls.key_file, tls.require)
.map_err(|e| format!("redshift: {e}"))?;

let server = Arc::new(Server::new(Config {
sql_addr: sql_addr.clone(),
api_addr: api_addr.clone(),
Expand All @@ -125,6 +130,7 @@ pub async fn run(
sql_backend,
translator,
events_enabled: true,
sql_tls,
}));

let sql_listener = TcpListener::bind(&sql_addr)
Expand Down
8 changes: 8 additions & 0 deletions services/redshift/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ path = "src/main.rs"
devcloud-s3 = { path = "../s3" }
# gzip COPY sources (pure-Rust backend; miniz_oxide is already in the tree).
flate2 = { version = "1", default-features = false, features = ["rust_backend"] }
# TLS on the SQL endpoint (pure-Rust rustls with the ring provider; no
# OpenSSL or CMake needed). Certificates come from files the user supplies.
tokio-rustls = { version = "0.26", default-features = false, features = ["ring", "tls12"] }
rustls-pki-types = { version = "1", features = ["std"] }
md-5 = "0.10"
sha2 = "0.10"
hmac = "0.12"
Expand All @@ -25,3 +29,7 @@ serde = { version = "1", features = ["derive"] }
# map marshaling. Do NOT enable preserve_order.
serde_json = "1"
tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros", "net", "io-util", "signal", "time", "sync"] }

[dev-dependencies]
# Test-only: issues a throwaway CA and server certificate for the TLS tests.
rcgen = { version = "0.13", default-features = false, features = ["ring", "pem"] }
1 change: 1 addition & 0 deletions services/redshift/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ pub mod snapshot;
pub mod sql_parse;
pub mod sql_predicates;
pub mod storage;
pub mod tls;
pub mod translator;

pub use backend::{SqlBackend, SqlTransaction};
Expand Down
Loading