Version: 2.0.0

Publish data from Amazon S3

Ingest objects from a bucket

warning

There are several ways to pass secrets into Tabsdata. See Secrets Management for more info if you are planning to have an LLM register secrets into Tabsdata's credential store.

Can you publish trips.parquet out of amazon s3

Listed 1 directory, called tabsdata 3 times (ctrl+o to expand)

The S3 publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them:

collection name
bucket
region
AWS keys

everything you need is in

Read 1 file, called tabsdata 2 times (ctrl+o to expand)

Both keys found. Reading back what is not secret: bucket my-bucket, region us-east-1, collection s3_src.

Called tabsdata 4 times (ctrl+o to expand)

Connection written to , and collection s3_src created from it in group sources with base_path /data. Connection check passed.

Called tabsdata 2 times (ctrl+o to expand)

The publisher, registered in s3_src:

@publisher(
  source=S3Src(paths=["trips.parquet"]),
  output_tables=["trips"],
)
def read_trips(trips: TableFramesSpec) -> TableFrameSpec:
  return trips[0]

Called tabsdata 6 times (ctrl+o to expand)

Execution committed, 0 failures.

s3_src/trips is live, holds 1,000,000 rows across 21 columns:

┌──────────────────┬──────────────┐
│      field       │    value     │
├──────────────────┼──────────────┤
│ collection       │ s3_src       │
├──────────────────┼──────────────┤
│ table            │ trips        │
├──────────────────┼──────────────┤
│ rows             │ 1,000,000    │
├──────────────────┼──────────────┤
│ columns          │ 21           │
├──────────────────┼──────────────┤
│ function version │ fv20260813-1 │
├──────────────────┼──────────────┤
│ execution        │ ep20260813-1 │
└──────────────────┴──────────────┘

note

Files of different formats in one paths list need no extra configuration. AUTO reads each file's extension separately, so a .csv and a .parquet in the same publisher both land correctly.

The case that does need work is objects whose extensions do not say what they are. format is one override for the whole source rather than one per path, so setting it reads every path as that format. Files that genuinely differ and cannot be recognised by name have to be split across publishers, one per format.

Ingest objects from a bucket​

Ingest objects from a bucket