OTR Transportation is a Chicago-based freight brokerage whose brokers depend on timely market, shipment, and logistics data to set competitive rates. OTR Transportation's growing customer base meant its brokers needed intelligent dashboards covering market demand, container shipments, and traffic footage to make better rate decisions, but the underlying data was scattered across multiple AWS accounts with no central way to query it. ASCENDING built a secure, centralized data lake on AWS Lake Formation that unified those sources and gave OTR's team self-service access to the analytics it needed.
BackgroundA Growing Freight Brokerage With Data Spread Across Multiple AWS Accounts
OTR Transportation, Inc., headquartered in Chicago, is a player in the freight transportation arrangement industry known for its experienced team of brokers spanning myriad industries. As OTR's customer base grew, the company wanted to develop intelligent dashboards that would let brokers pull in data on market-demanded areas, container shipments, and traffic footage so stakeholders could make better rate decisions based on real market trends.
OTR needed centralized data storage for the large volume of data generated by its multiple ongoing data acquisition processes, with that same data lake serving as the foundation for future analytics, reporting, and machine learning applications.
The ChallengeFragmented Data Across Three AWS Accounts With No Self-Service Access
The challenge is a data environment that outgrew OTR's manual processes for governance, access, and reporting. OTR's data was growing faster than its existing processes could organize or make useful.
- More than 10 GB of user log and API interaction data was being generated every day, creating a data governance and management bottleneck.
- OTR had no way to run ad hoc queries or research existing data, and instead depended on data specialists to manually finalize Excel reports and metrics.
- Data originated from three separate AWS application accounts, with no centralized, secure way to bring it together.
- Any solution needed to scale with OTR's continued growth while remaining secure and properly governed.
An AWS Advanced Consulting Partner for Data Lake Architecture
ASCENDING is an AWS Certified Advanced Consulting Partner with deep experience in serverless architecture, ETL, data visualization, and AWS infrastructure automation, having delivered similar data lake projects for organizations at various scales. Previous clients had described the ASCENDING team as responsive, friendly, and easy to communicate with, which, combined with that technical track record, made ASCENDING OTR's choice to take on the assignment.
The SolutionA Centralized Data Lake Built on AWS Lake Formation
The solution is a centralized data lake architecture built on AWS Lake Formation that unifies data from every AWS account into one governed source. Since OTR's data sources were generated across three different AWS application accounts, ASCENDING designed a centralized AWS account to house the data lake and unify everything in one place.
- AWS Glue and Amazon Kinesis moved data from third-party sources, NoSQL databases, and relational databases into Source S3 buckets that served as the data source layer.
- S3 policies enabled secure cross-account access, letting data flow from each application account into the central data lake account.
- Glue Jobs securely transformed and enriched the data as it moved into the data lake, landing in a dedicated "Landing Bucket."
- Once cataloged in AWS Lake Formation, downstream applications like Amazon Athena and Amazon QuickSight were granted access for further analysis and dashboarding.
A Secure, Self-Service Data Lake for OTR's Growing Business
The outcome is a centralized, governed data lake that gives OTR's team self-service analytics without compromising security. ASCENDING delivered the data lake infrastructure on OTR's timeline, giving the company a centralized, secure foundation for its growing data needs.
- Resolved OTR's core big data challenges around data governance, scalable infrastructure, and rapid data acquisition.
- Created a centralized, secured data lake for both internal and external data sources, with access governed by AWS Lake Formation's permissions model.
- Gave staff across the organization the ability to access data with proper permissions and constraints, rather than routing every request through data specialists.
- Included ongoing system maintenance, performance monitoring, and access monitoring as part of the engagement.
- Saved OTR significant time and cost on repeated manual work, freeing the team to focus on turning data into business value.
Built with AWS Data Lake and Analytics Services
Frequently Asked Questions
How long does it take to build a centralized data lake on AWS Lake Formation?
Timeline is a phased engagement that starts with mapping data sources across AWS accounts, then layers in ingestion, cataloging, and governance before self-service tools are opened up to end users — OTR's build followed this same sequence, from source S3 buckets through Lake Formation cataloging.
How is data kept secure when it originates from multiple AWS accounts?
Security is enforced through cross-account S3 policies and a centrally managed permissions model, so each application account can contribute data to the shared lake without exposing it outside of approved, governed access paths.
Can business users query the data lake without going through IT or data specialists?
Yes — once data is cataloged, business users can run ad hoc queries directly in Amazon Athena or build their own dashboards in Amazon QuickSight, removing the bottleneck of routing every request through a data specialist.
