I am a technology journalist with three decades of press experience. Primarily I work as a news analysis writer dedicated to a software application development ‘beat’. I have spent much of the last twenty years focusing on open source, analytics & data science, cloud computing, mobile devices & data management.
Pipelines run the world. Whether it is petroleum, gas, hydrogen, sewage or some form of slurry or indeed the cabling pipelines that help to create the Internet, pipelines are pretty much at the heart of everything. As we move from the physical world to the digital fabric now underpinning society, we also talk about data pipelines.
Some data transformation techniques change formats like JSON into tables, while others take tabular data and flatten or ‘denormalize’ it. Both allow the analytical database to process the data more easily and are done in a series of steps to extract, load and transform it.
“In addition to joins, the need for aggregation is another major reason organizations turn to transformation pipelines,” explained CelerData’s Oliver. “Data is often pre-aggregated for summation, grouping, counts etc. Databases with efficient cost-based optimizers that support materialized views can avoid the pipeline by performing ‘Extract, Load, Transform’ instead of traditional ‘Extract, Transform, Load’ .
“By using a newer query engine and data lakehouse technologies, it's possible to avoid transformation pipelines entirely. There will still be sources and destinations, but no independent stages. Complex queries may not be as efficient as pre-flattened data, but the architecture will be simpler and there is no cost in terms of compute, memory, or storage to pre-transform data that is ultimately left unqueried. This ‘pipeline free’ architecture also ensures that data is fresher,” said Oliver.
Deutschland Neuesten Nachrichten, Deutschland Schlagzeilen
Similar News:Sie können auch ähnliche Nachrichten wie diese lesen, die wir aus anderen Nachrichtenquellen gesammelt haben.
1st section of massive NYC landfill-to-park transformation project opens to the publicIn 2008, a New York City project began to transform a massive landfill site on Staten Island into a park. On Sunday, the first section of Freshkills Park opened to the public.
Weiterlesen »
Amazing Star Wars Art Shows Anakin Skywalker's Transformation Into Darth VaderDarth Vader’s true identity resurfaces in this stunning fan art that brings Anakin back from behind his mask in a most interesting way.
Weiterlesen »
Giants vs. Bills: Preview, prediction, what to watch forAn inside look at Sunday’s Giants-Bills NFL Week 6 matchup in Orchard Park, N.Y.:
Weiterlesen »
Jets vs. Eagles: Preview, prediction, what to watch forAn inside look at Sunday’s Jets-Eagles NFL Week 6 matchup at MetLife Stadium:
Weiterlesen »
Inside Texas’ explosion of uncertified new teachersA Dallas Morning News analysis of statewide hiring data from the 2022-23 school year reveals the extent to which the reliance on uncertified new educators...
Weiterlesen »
A look inside Tucson's new sewing school and charity factoryRyan comes to the Sonoran Desert from California’s Central Coast after working as a reporter, sports anchor and weather forecaster at KEYT in Santa Barbara.
Weiterlesen »