Zeljko Obrenovic' Technology Catalog

A cloud database is a database that typically runs on a cloud computing platform, and access to the database is provided as-a-service. Database services take care of scalability and high availability of the database. Database services make the underlying software-stack transparent to the user.

There are two primary methods to run a database in a cloud:

Databases

Relational Databases (5)

Key-Value Databases (4)

Document Databases (3)

In-Memory Databases (4)

Wide Column Databases (2)

Graph Databases (1)

Time-Series Databases (1)

Ledger Databases (1)

Data Warehouses (3)

ETL (3)

Advanced Analytics Processing (3)

Search as a Service (4)

Data Lake as a Service (3)

Databases

A database is a collection of data that is organized so that it can be easily accessed, managed and updated.

Relational Databases (5)

SQL, transactions, and strong consistency at scale.

[AWS] Relational Database Service (RDS)
A distributed relational database service, a web service running "in the cloud" designed to simplify the setup, operation, and scaling of a relational database for use in applications.
Amazon Relational Database Service (or Amazon RDS) is a distributed relational database service by Amazon Web Services (AWS). It is a web service running "in the cloud" designed to simplify the setup, operation, and scaling of a relational database for use in applications. Administration processes like patching the database software, backing up databases and enabling point-in-time recovery are managed automatically. Scaling storage and compute resources can be performed by a single API call as AWS does not offer an ssh connection to RDS instances.

Source: https://www.percona.com/blog/wp-content/uploads/2018/07/RDS_Multi-AZ_1.png

Source: https://d2908q01vomqb2.cloudfront.net/fc074d501302eb2b93e2554793fcaf50b3bf7291/2018/09/13/PeopleSoft_migrate_AWS.png

Links:
- aws.amazon.com/rds
- en.wikipedia.org/wiki/Amazon_Relational_Database_Service
[AWS] Aurora
A MySQL and PostgreSQL-compatible relational database built for the cloud, that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases.
Amazon Aurora is a MySQL and PostgreSQL-compatible relational database built for the cloud, that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases.

Amazon Aurora is up to five times faster than standard MySQL databases and three times faster than standard PostgreSQL databases. It provides the security, availability, and reliability of commercial databases at 1/10th the cost. Amazon Aurora is fully managed by Amazon Relational Database Service (RDS), which automates time-consuming administration tasks like hardware provisioning, database setup, patching, and backups.

Amazon Aurora features a distributed, fault-tolerant, self-healing storage system that auto-scales up to 64TB per database instance. It delivers high performance and availability with up to 15 low-latency read replicas, point-in-time recovery, continuous backup to Amazon S3, and replication across three Availability Zones (AZs).

Source: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/images/AuroraArch001.png

Source: https://d1.awsstatic.com/webteam/category-pages/Databases/Category-Page-Diagram_Datbases-on-AWS_Enterprise-Applications.fa44910d3054c6b9f6fc5140fb77b0e056dc8df7.png

Source: https://d1.awsstatic.com/webteam/category-pages/Databases/category-page-diagram_database_gaming.de2623720c885acfe788a9170db1837129758d8a.png

Links:
- aws.amazon.com/rds/aurora
- en.wikipedia.org/wiki/Amazon_Aurora
[Azure] SQL Server
A managed cloud database (formerly SQL Azure, SQL Server Data Services, SQL Services, and Windows Azure SQL Database).
Microsoft Azure SQL Database (formerly SQL Azure, SQL Server Data Services, SQL Services, and Windows Azure SQL Database) is a managed cloud database (PaaS) provided as part of Microsoft Azure. A cloud database is a database that runs on a cloud computing platform, and access to it is provided as a service. Managed database services take care of scalability, backup, and high availability of the database. Azure SQL Database is a managed database service which is different from AWS RDS which is a container service.

Microsoft Azure SQL Database includes built-in intelligence that learns app patterns and adapts to maximize performance, reliability, and data protection.

Source: https://docs.microsoft.com/en-us/azure/sql-database/media/sql-database-technical-overview/deployment-options.png

Source: https://docs.microsoft.com/en-us/azure/sql-database/media/sql-database-what-is-a-dtu/sqldb_elastic_pools.png

Source: https://docs.microsoft.com/en-us/azure/sql-database/media/sql-database-metrics-diag-logging/architecture.png

Source: https://docs.microsoft.com/en-us/azure/sql-database/media/sql-database-paas-vs-sql-server-iaas/sqliaas_sql_server_cloud_continuum.png

Links:
- azure.microsoft.com/en-us/services/sql-database
- en.wikipedia.org/wiki/Microsoft_Azure_SQL_Database
[Google Cloud] SQL
Fully managed relational database service for MySQL, PostgreSQL, and SQL server.
Cloud SQL is fully compatible with applications using MySQL, PostgreSQL, and SQL Server. You can connect with nearly any application, anywhere in the world. Cloud SQL automates backups, replication, and failover to ensure your database is reliable, highly available, and flexible to your performance needs.

Source: https://storage.googleapis.com/gweb-cloudblog-publish/images/image1d95y.max-700x700.PNG

Source: https://cloud.google.com/images/sql/one-market_2x.png

Links:
- cloud.google.com/sql
[Google Cloud] Spanner
A NewSQL database providing global transactions, strongly consistent reads, and automatic multi-site replication and failover.
Spanner is a NewSQL* database developed by Google. Spanner is a globally distributed database service and storage solution. It provides features such as global transactions, strongly consistent reads, and automatic multi-site replication and failover. [*NewSQL is a class of relational database management systems that seek to provide the scalability of NoSQL systems for online transaction processing (OLTP) workloads while maintaining the ACID guarantees of a traditional database system.]

Source: https://cloud.google.com/solutions/images/data-lifecycle-1.svg

Source: https://cloud.google.com/solutions/images/data-lifecycle-2.svg

Links:
- cloud.google.com/spanner
- en.wikipedia.org/wiki/Spanner_(database)

Key-Value Databases (4)

Optimized for common access patterns, typically to store and retrieve large volumes of data. These databases deliver quick response times, even in extreme volumes of concurrent requests.

[AWS] DynamoDB
A fully managed proprietary NoSQL database service that supports key-value and document data structures.
Amazon DynamoDB is a fully managed proprietary NoSQL database service that supports key-value and document data structures and is offered by Amazon.com as part of the Amazon Web Services portfolio. DynamoDB exposes a similar data model to and derives its name from Dynamo, but has a different underlying implementation. Dynamo had a multi-master design requiring the client to resolve version conflicts and DynamoDB uses synchronous replication across multiple data centers for high durability and availability. DynamoDB was announced by Amazon CTO Werner Vogels on January 18, 2012, and is presented as an evolution of Amazon SimpleDB solution.

Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/SQLtoNoSQL.png

Source: https://d1.awsstatic.com/diagrams/product-page-diagrams/Use_Case_WebApplications.2139ddbc8a84f5564ee5846995f28c88e9db5c2d.png

Source: https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/Product-Page-Diagram_AppSync_DynamoDB_Mobile-Backends.45005baf263c67b8ab0d1467e535eb538f0b93de.png

Source: https://d1.awsstatic.com/diagrams/product-page-diagrams/Use_Case_StreamProcessing.8464961e382ff17c57750f7b1dc23c1ef7e4c233.png

Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/HowItWorksPeople.png

Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/HowItWorksGenreAlbumTitle.png

Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/images/HowItWorksStreams.png

Links:
- aws.amazon.com/dynamodb
- en.wikipedia.org/wiki/Amazon_DynamoDB
[Google Cloud] Datastore (deprecated)
Cloud Datastore is a highly-scalable NoSQL database for your web and mobile applications.
Cloud Datastore is a highly scalable NoSQL database for your applications. Cloud Datastore automatically handles sharding and replication, providing you with a highly available and durable database that scales automatically to handle your applications' load. Cloud Datastore provides a myriad of capabilities such as ACID transactions, SQL-like queries, indexes, and much more.
Links:
- cloud.google.com/datastore
[Google Cloud] Firestore
Store cloud-native app data at global scale.
Cloud Firestore is a fast, fully managed, serverless, cloud-native NoSQL document database that simplifies storing, syncing, and querying data for your mobile, web, and IoT apps at global scale. Its client libraries provide live synchronization and offline support, while its security features and integrations with Firebase and Google Cloud Platform (GCP) accelerate building truly serverless apps.

Source: https://cloud.google.com/firestore/docs/images/structure-data.png

Source: https://storage.googleapis.com/gweb-cloudblog-publish/images/expenses_app.max-700x700.png

Source: https://storage.googleapis.com/gweb-cloudblog-publish/images/cloud_functions.max-600x600.png

Source: https://codelabs.developers.google.com/codelabs/modern-data-pipeline-firestore-bigquery-dataflow-templates/img/707cc6119e040c75.png

Links:
- cloud.google.com/firestore
[Azure] Table Storage
A service that stores structured NoSQL data in the cloud, providing a key/attribute store with a schemaless design.
Azure Table storage is a service that stores structured NoSQL data in the cloud, providing a key/attribute store with a schemaless design. Because Table storage is schemaless, it's easy to adapt your data as the needs of your application evolve. Access to Table storage data is fast and cost-effective for many types of applications, and is typically lower in cost than traditional SQL for similar volumes of data.

Source: https://docs.microsoft.com/en-us/azure/includes/media/storage-table-concepts-include/table1.png

Links:
- azure.microsoft.com/en-us/services/storage/tables

Document Databases (3)

Store semistructured data as JSON-like documents.

[AWS] DocumentDB
Fast, scalable, highly available MongoDB-compatible database service.
Amazon DocumentDB (with MongoDB compatibility) is a fast, scalable, highly available, and fully managed document database service that supports MongoDB workloads. As a document database, Amazon DocumentDB makes it easy to store, query, and index JSON data.

Amazon DocumentDB is a non-relational database service designed from the ground-up to give you the performance, scalability, and availability you need when operating mission-critical MongoDB workloads at scale. In Amazon DocumentDB, the storage and compute are decoupled, allowing each to scale independently, and you can increase the read capacity to millions of requests per second by adding up to 15 low latency read replicas in minutes, regardless of the size of your data.

Amazon DocumentDB is designed for 99.99% availability and replicates six copies of your data across three AWS Availability Zones (AZs). You can use AWS Database Migration Service (DMS) for free (for six months) to easily migrate your on-premises or Amazon Elastic Compute Cloud (EC2) MongoDB databases to Amazon DocumentDB with virtually no downtime.

Source: https://d1.awsstatic.com/AWS%20Databases/Product-Page-Diagram_Chimera_Database-migration@2x.96269f40b3c0476e5137a87fdc47de76e39f3b57.png

Source: https://d1.awsstatic.com/AWS%20Databases/Product-Page-Diagram_DocumentDB_HIW_REVIEW.78f18079b219164cfe7a00a29be24e2b96264e8a.png

Links:
- aws.amazon.com/documentdb
[Azure] Cosmos DB
Globally-distributed, multi-model database service "for managing data at planet-scale".
Azure Cosmos DB is Microsoft's proprietary globally-distributed, multi-model database service "for managing data at planet-scale" launched in May 2017. It is schema-agnostic, horizontally scalable and generally classified as a NoSQL database.

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/introduction/azure-cosmos-db.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/relational-or-nosql/backend-scaled.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/relational-or-nosql/order-orderdetails.jpg

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/relational-or-nosql/graph.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/distribute-data-globally/deployment-topology.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/consistency-levels/five-consistency-levels.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/consistency-levels-across-apis/consistency-model-mapping-mongodb.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/scaling-throughput/request-unit-charge-of-read-and-write-operations.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/global-dist-under-the-hood/distributed-system-topology.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/global-dist-under-the-hood/distribution-of-resource-partitions.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/global-dist-under-the-hood/dynamic-overlay-of-resource-partitions.png

Source: https://docs.microsoft.com/en-us/azure/cosmos-db/media/globally-distributed-transactional-analytical-storage/storage-engines-api-mapping.png

Links:
[Google Cloud] Bigtable
A petabyte-scale, fully managed NoSQL database service for large analytical and operational workloads.
Bigtable is a compressed, high performance, proprietary data storage system built on Google File System, Chubby Lock Service, SSTable (log-structured storage like LevelDB) and a few other Google technologies. On May 6, 2015, a public version of Bigtable was made available as a service. Bigtable also underlies Google Cloud Datastore, which is available as a part of the Google Cloud Platform.

Source: https://cloud.google.com/bigtable/img/bigtable-example.svg

Source: https://cloud.google.com/bigtable/img/bigtable-architecture.svg

Links:
- cloud.google.com/bigtable
- en.wikipedia.org/wiki/Bigtable

In-Memory Databases (4)

Used for applications that require real-time access to data.

[AWS] ElastiCache for Redis
Redis compatible in-memory data store built for the cloud. Power real-time applications with sub-millisecond latency.
Amazon ElastiCache for Redis is a blazing fast in-memory data store that provides sub-millisecond latency to power internet-scale real-time applications. Built on open-source Redis and compatible with the Redis APIs, ElastiCache for Redis works with your Redis clients and uses the open Redis data format to store your data. Your self-managed Redis applications can work seamlessly with ElastiCache for Redis without any code changes. ElastiCache for Redis combines the speed, simplicity, and versatility of open-source Redis with manageability, security, and scalability from Amazon to power the most demanding real-time applications in Gaming, Ad-Tech, E-Commerce, Healthcare, Financial Services, and IoT.

Source: https://d1.awsstatic.com/webteam/category-pages/Databases/category-page-diagram_database_caching.959d5429c58d7a59fa08e3484ff7d43b4881ebdc.png

Source: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/images/ElastiCache-Caching.png

Source: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/images/ElastiCache-Redis-Gaming.png

Source: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/images/ElastiCache-Redis-PubSub.png

Source: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/images/ElastiCache-Redis-Analytics.png

Links:
- aws.amazon.com/elasticache/redis
[AWS] ElastiCache for Memcached
Managed, Memcached-compatible, in-memory store. Sub-millisecond latency to power real-time applications.
Amazon ElastiCache for Memcached is a Memcached-compatible in-memory key-value store service that can be used as a cache or a data store. It delivers the performance, ease-of-use, and simplicity of Memcached. ElastiCache for Memcached is fully managed, scalable, and secure - making it an ideal candidate for use cases where frequently accessed data must be in-memory. It is a popular choice for use cases such as Web, Mobile Apps, Gaming, Ad-Tech, and E-Commerce.

Source: https://d1.awsstatic.com/webteam/category-pages/Databases/category-page-diagram_database_open-source.b19513e06bb99e0cf4a9f98b389f3c891ebb2843.png

Links:
- aws.amazon.com/elasticache/memcached
[Google Cloud] Memorystore
Fully managed in-memory data store service for Redis and Memcached.
Memorystore provides a fully managed in-memory data store service built on scalable, secure, and highly available infrastructure managed by Google. Use Memorystore to build application caches that provide sub-millisecond data access. Memorystore is compatible with the Redis and Memcached open source protocol, allowing easy migration with zero code changes.

Source: https://storage.googleapis.com/gcp-community/tutorials/memorystore-oc/image4.png

Source: https://image.slidesharecdn.com/gopalashokkarthithyagarajan-180530173854/95/redisconf18-redis-on-google-cloud-platform-25-638.jpg?cb=1527701979

Links:
- cloud.google.com/memorystore
[Azure] Cache for Redis
Fully managed, open source–compatible in-memory data store to power fast, scalable applications.
Azure Cache for Redis achieves superior throughput and latency performance by storing data in memory instead of on disk. It consistently serves read and write requests within single-digit milliseconds, delivering exceedingly fast cache operations to scale data tiers as application loads increase.

Source: https://docs.microsoft.com/en-us/azure/architecture/solution-ideas/media/data-cache-with-redis-cache.png

Links:
- azure.microsoft.com/en-us/services/cache

Wide Column Databases (2)

Uses tables, rows, and columns, but unlike a relational database, the names and format of the columns can vary from row to row in the same table.

[AWS] Managed Apache Cassandra Service
A scalable, highly available, and managed Apache Cassandra–compatible database service.
Amazon Managed Apache Cassandra Service is a scalable, highly available, and managed Apache Cassandra–compatible database service. With Amazon Managed Cassandra Service, you can run your Cassandra workloads on AWS using the same Cassandra application code and developer tools that you use today. You don’t have to provision, patch, or manage servers, and you don’t have to install, maintain, or operate software. Amazon Managed Cassandra Service is serverless, so you pay for only the resources you use and the service automatically scales tables up and down in response to application traffic. You can build applications that serve thousands of requests per second with virtually unlimited throughput and storage.

Source: https://dd25om75gc08l.cloudfront.net/20200421160100136/images/mcs.png

Source: https://docs.aws.amazon.com/mcs/latest/devguide/images/cassandra-hi-level.png

Source: https://docs.aws.amazon.com/mcs/latest/devguide/images/mcs-hi-level.png

Links:
- aws.amazon.com/mcs
[AWS] Keyspaces
A scalable, highly available, and managed Apache Cassandra–compatible database service.
Amazon Keyspaces (for Apache Cassandra) is a scalable, highly available, and managed Apache Cassandra–compatible database service. With Amazon Keyspaces, you can run your Cassandra workloads on AWS using the same Cassandra application code and developer tools that you use today. You don’t have to provision, patch, or manage servers, and you don’t have to install, maintain, or operate software. Amazon Keyspaces is serverless, so you pay for only the resources you use and the service can automatically scale tables up and down in response to application traffic. You can build applications that serve thousands of requests per second with virtually unlimited throughput and storage. Amazon Keyspaces gives you the performance, elasticity, and enterprise features you need to operate business-critical Cassandra workloads at scale.

Source: https://docs.aws.amazon.com/keyspaces/latest/devguide/images/cassandra-hi-level.png

Source: https://docs.aws.amazon.com/keyspaces/latest/devguide/images/mcs-hi-level.png

Source: https://docs.aws.amazon.com/keyspaces/latest/devguide/images/auto-scaling.png

Links:
- aws.amazon.com/keyspaces

Graph Databases (1)

[AWS] Neptune
A fast, reliable, fully managed graph database service that makes it easy to build and run applications that work with highly connected datasets.
Amazon Neptune is a fast, reliable, fully managed graph database service that makes it easy to build and run applications that work with highly connected datasets. The core of Amazon Neptune is a purpose-built, high-performance graph database engine optimized for storing billions of relationships and querying the graph with milliseconds latency. Amazon Neptune supports popular graph models Property Graph and W3C's RDF, and their respective query languages Apache TinkerPop Gremlin and SPARQL, allowing you to easily build queries that efficiently navigate highly connected datasets. Neptune powers graph use cases such as recommendation engines, fraud detection, knowledge graphs, drug discovery, and network security.

Amazon Neptune is highly available, with read replicas, point-in-time recovery, continuous backup to Amazon S3, and replication across Availability Zones. Neptune is secure with support for HTTPS encrypted client connections and encryption at rest. Neptune is fully managed, so you no longer need to worry about database management tasks such as hardware provisioning, software patching, setup, configuration, or backups.

Source: https://d1.awsstatic.com/Products/product-name/diagrams/product-page-diagram_Neptune.85fa1536c4a013c5f291b930985032d693dd8111.png

Source: https://d1.awsstatic.com/product-marketing/Neptune/Neptune-Diagram_social-v3.20da6a2fc4f25a212e995894b52e721b403b58ba.png

Source: https://d1.awsstatic.com/product-marketing/Neptune/Neptune-Diagram_recommendation-relationships.ba1ed4bbfe14f9b810f64bb0b16065daa3006eed.png

Source: https://d1.awsstatic.com/product-marketing/Neptune/Neptune-Diagram_Fraud-Detection.b1edc3a8b6f9a0481d9431db1f4d4da78c33c487.png

Source: https://d1.awsstatic.com/product-marketing/Neptune/Neptune-Diagram_knowledge-graph.22076f2eae8aa3e7f3c6c7e7b11a181dfe1b4002.png

Source: https://d1.awsstatic.com/product-marketing/Neptune/Neptune-Diagram_life-sciences.429803b3bdbc184b3e0a52e8f49a260bc09ded65.png

Source: https://d1.awsstatic.com/product-marketing/Neptune/Neptune-Diagram_network-security.ea548ac6f4ce25134d1c9559f957350af2071dfa.png

Links:
- aws.amazon.com/neptune

Time-Series Databases (1)

Collect, synthesize, and derive insights from data that changes over time and with queries spanning time intervals.

[AWS] Timestream
A fast, scalable, fully managed time series database service for IoT and operational applications that makes it easy to store and analyze trillions of events per day at 1/10th the cost of relational databases.
Amazon Timestream is a fast, scalable, fully managed time series database service for IoT and operational applications that makes it easy to store and analyze trillions of events per day at 1/10th the cost of relational databases. Driven by the rise of IoT devices, IT systems, and smart industrial machines, time-series data — data that measures how things change over time — is one of the fastest growing data types. Time-series data has specific characteristics such as typically arriving in time order form, data is append-only, and queries are always over a time interval. While relational databases can store this data, they are inefficient at processing this data as they lack optimizations such as storing and retrieving data by time intervals. Timestream is a purpose-built time series database that efficiently stores and processes this data by time intervals. With Timestream, you can easily store and analyze log data for DevOps, sensor data for IoT applications, and industrial telemetry data for equipment maintenance. As your data grows over time, Timestream’s adaptive query processing engine understands its location and format, making your data simpler and faster to analyze. Timestream also automates rollups, retention, tiering, and compression of data, so you can manage your data at the lowest possible cost. Timestream is serverless, so there are no servers to manage. It manages time-consuming tasks such as server provisioning, software patching, setup, configuration, or data retention and tiering, freeing you to focus on building your applications.
Links:
- aws.amazon.com/timestream

Ledger Databases (1)

A centralized and trusted authority to maintain a scalable, immutable, and cryptographically verifiable record of transactions for every application.

[AWS] Quantum Ledger Database (QLDB)
Fully managed ledger database that provides a transparent, immutable, and cryptographically verifiable transaction log. Owned by a central trusted authority.
Amazon QLDB is a fully managed ledger database that provides a transparent, immutable, and cryptographically verifiable transaction log ‎owned by a central trusted authority. Amazon QLDB tracks each and every application data change and maintains a complete and verifiable history of changes over time.

Ledgers are typically used to record a history of economic and financial activity in an organization. Many organizations build applications with ledger-like functionality because they want to maintain an accurate history of their applications' data, for example, tracking the history of credits and debits in banking transactions, verifying the data lineage of an insurance claim, or tracing movement of an item in a supply chain network. Ledger applications are often implemented using custom audit tables or audit trails created in relational databases. However, building audit functionality with relational databases is time-consuming and prone to human error. It requires custom development, and since relational databases are not inherently immutable, any unintended changes to the data are hard to track and verify. Alternatively, blockchain frameworks, such as Hyperledger Fabric and Ethereum, can also be used as a ledger. However, this adds complexity as you need to set-up an entire blockchain network with multiple nodes, manage its infrastructure, and require the nodes to validate each transaction before it can be added to the ledger.

Amazon QLDB is a new class of database that eliminates the need to engage in the complex development effort of building your own ledger-like applications. With QLDB, your data’s change history is immutable – it cannot be altered or deleted – and using cryptography, you can easily verify that there have been no unintended modifications to your application’s data. QLDB uses an immutable transactional log, known as a journal, that tracks each application data change and maintains a complete and verifiable history of changes over time. QLDB is easy to use because it provides developers with a familiar SQL-like API, a flexible document data model, and full support for transactions. QLDB is also serverless, so it automatically scales to support the demands of your application. There are no servers to manage and no read or write limits to configure. With QLDB, you only pay for what you use.

Source: https://d1.awsstatic.com/r2018/h/99Product-Page-Diagram_AWS-Quantum.f03953678ba33a2d1b12aee6ee530e45507e7ac9.png

Links:
- aws.amazon.com/qldb

Data Warehouses (3)

A data warehouse (DW or DWH), also known as an enterprise data warehouse (EDW), is a centralized repository of integrated data from one or more disparate sources.

[AWS] Redshift
The most popular and fastest cloud data warehouse.
With Redshift you can query petabytes of structured and semi-structured data across your data warehouse and your data lake using standard SQL. Redshift lets you easily save the results of your queries back to your S3 data lake using open formats like Apache Parquet to further analyze from other analytics services like Amazon EMR, Amazon Athena, and Amazon SageMaker.

Source: https://d1.awsstatic.com/re19/Redshift/Product-Page-Diagram_Redshift_Data-Lake-Integration@2x.b04dff434226a5b3e5e38159bbf610ee617a5655.png

Source: https://d1.awsstatic.com/redshift/Redshift_AQUA_Simple@2x.e0b6d21f14418792ed59ca8aabd8d68432e9d6c5.png

Links:
- aws.amazon.com/redshift
Google BigQuery
A serverless, highly scalable, and cost-effective cloud data warehouse designed to help you make informed decisions quickly, so you can transform your business with ease.
Accelerate time-to-value with a fully managed and serverless cloud data warehouse that is easy to set up and manage and doesn’t require a database administrator. Jump-start data analysis cost effectively and uncover meaningful insights to stay competitive.

Source: https://cloudx-bricks-prod-bucket.storage.googleapis.com/f88da9b0ea355143133a0856fd2e3e5ec4f4f42649d42a230bcb0601e7599694.svg

Source: https://cloudx-bricks-prod-bucket.storage.googleapis.com/6fb126b82a6572969f882ca511db6565eabc3b933e46a0d08e7c3aab1a916046.svg

Links:
- cloud.google.com/bigquery
[Azure] Synapse Analytics (formerly SQL Data Warehouse)
Limitless analytics service with unmatched time to insight.
Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

Source: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/media/sql-data-warehouse-overview-what-is/data-warehouse-solution.png

Source: https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/data/images/enterprise-bi-synapse.png

Source: https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/data/images/enterprise-bi-sqldw-pipeline.png

Source: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/media/massively-parallel-processing-mpp-architecture/massively-parallel-processing-mpp-architecture.png

Source: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/media/massively-parallel-processing-mpp-architecture/hash-distributed-table.png

Source: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/media/massively-parallel-processing-mpp-architecture/replicated-table.png

Links:
- azure.microsoft.com/en-us/services/synapse-analytics

ETL (3)

Extract, transform, load (ETL) is the general procedure of copying data from one or more sources into a destination system which represents the data differently from the source(s) or in a different context than the source(s).

[AWS] Glue
Simple, flexible, and cost-effective ETL.
AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once cataloged, your data is immediately searchable, queryable, and available for ETL.

Source: https://d1.awsstatic.com/Products/product-name/diagrams/product-page-diagram_Glue_Queries-Against-an-Amazo-S3-Data-Lake.0b355cc000b96c506babaa28fa8d7b4211b50c6f.png

Source: https://d1.awsstatic.com/Products/product-name/diagrams/product-page-diagram_Glue_Analyze-Log-Data-in-Data-Warehouse.5b156ca0d97dc67c56f12b6df690c5928e33a2a4.png

Source: https://d1.awsstatic.com/Products/product-name/diagrams/product-page-diagram_Glue_Unified-View-of-Data-Across-Multiple-Data-Stores.0aa670c04b8b8f68f36fb2f6916cd0bf1b12f43a.png

Source: https://d1.awsstatic.com/Products/product-name/diagrams/product-page-diagram_Glue_Event-driven-ETL-Pipelines.e24d59bb79a9e24cdba7f43ffd234ec0482a60e2.png

Source: https://d1.awsstatic.com/screenshots/glue/screenshot-glue-step2-etl-generation4.6925ce200fa9713b20d1236d7079b539ad204c6f.png

Links:
- aws.amazon.com/glue
[Google Cloud] Data Fusion
Fully managed, code-free data integration at any scale.
Cloud Data Fusion is a fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. With a graphical interface and a broad open source library of preconfigured connectors and transformations, Cloud Data Fusion shifts an organization’s focus away from code and integration to insights and action.

Source: https://cloud.google.com/data-fusion/docs/images/quickstart/studio-deploy.png

Source: https://cloud.google.com/data-fusion/docs/images/quickstart/studio-details.png

Source: https://cloud.google.com/data-fusion/docs/images/quickstart/details-run.png

Links:
- cloud.google.com/data-fusion
[Azure] Data Factory
Hybrid data integration service that simplifies ETL at scale.
Integrate data silos with Azure Data Factory, a service built for all data integration needs and skill levels. Easily construct ETL and ELT processes code-free within the intuitive visual environment, or write your own code. Visually integrate data sources using more than 90+ natively built and maintenance-free connectors at no added cost. Focus on your data—the serverless integration service does the rest.

Source: https://azurecomcdn.azureedge.net/cvt-f5458f331e589f4df3ee1444da812347560174b9cfa47ae530103d627388294d/images/page/services/data-factory/img-valprop2.jpg

Source: https://azurecomcdn.azureedge.net/cvt-f5458f331e589f4df3ee1444da812347560174b9cfa47ae530103d627388294d/images/page/services/data-factory/img-valprop1.jpg

Links:
- azure.microsoft.com/en-us/services/data-factory

Advanced Analytics Processing (3)

Advanced Analytics is the autonomous or semi-autonomous examination of data or content using sophisticated techniques and tools, typically beyond those of traditional business intelligence (BI), to discover deeper insights, make predictions, or generate recommendations.

[AWS] Elastic MapReduce (EMR)
Easily run and scale Apache Spark, Hadoop, HBase, Presto, Hive, and other big data frameworks.
Amazon EMR is the industry leading cloud-native big data platform for processing vast amounts of data quickly and cost-effectively at scale. Using open source tools such as Apache Spark, Apache Hive, Apache HBase, Apache Flink, Apache Hudi (Incubating), and Presto, coupled with the dynamic scalability of Amazon EC2 and scalable storage of Amazon S3, EMR gives analytical teams the engines and elasticity to run Petabyte-scale analysis for a fraction of the cost of traditional on-premises clusters. EMR gives teams the flexibility to run use cases on single-purpose, short lived clusters that automatically scale to meet demand, or on long running highly available clusters using the new multi-master deployment mode. If you have existing on-premises deployments of open source tools such as Apache Spark and Apache Hive, you can also run EMR clusters on AWS Outposts, giving you both the ability to scale out on-premises via Outposts or in the cloud.

Source: https://docs.aws.amazon.com/emr/latest/ManagementGuide/images/cluster-node-types.png

Source: https://docs.aws.amazon.com/emr/latest/ManagementGuide/images/step-sequence.png

Source: https://docs.aws.amazon.com/emr/latest/ManagementGuide/images/step-sequence-failed.png

Links:
- aws.amazon.com/emr
Google Dataflow
Fast, unified stream and batch data processing.
Dataflow is a fully managed streaming analytics service that minimizes latency, processing time, and cost through autoscaling and batch processing. With its serverless approach to resource provisioning and management, you have access to virtually limitless capacity to solve your biggest data processing challenges, while paying only for what you use.

Source: https://miro.medium.com/max/3250/1*FDttEYL6RFn09aYWBbN5Rg.png

Source: https://storage.googleapis.com/gweb-cloudblog-publish/images/Market_data_on_cloud.max-1400x1400.png

Source: https://miro.medium.com/max/3296/1*OaVwGy1MNotrx3Oi6a9Zeg.png

Source:

Links:
- cloud.google.com/dataflow
[Azure] HD Insight
Easy, cost-effective, enterprise-grade service for open source analytics.
Easily run popular open source frameworks—including Apache Hadoop, Spark, and Kafka—using Azure HDInsight, a cost-effective, enterprise-grade service for open source analytics. Effortlessly process massive amounts of data and get all the benefits of the broad open source ecosystem with the global scale of Azure.

Source: https://docs.microsoft.com/en-us/azure/hdinsight/hadoop/media/apache-hadoop-introduction/hdinsight-architecture-data-warehouse.png

Source: https://docs.microsoft.com/en-us/azure/hdinsight/hadoop/media/apache-hadoop-introduction/hdinsight-architecture-iot.png

Source: https://docs.microsoft.com/en-us/azure/hdinsight/hadoop/media/apache-hadoop-introduction/hdinsight-architecture-data-science.png

Source: https://docs.microsoft.com/en-us/azure/hdinsight/hadoop/media/apache-hadoop-introduction/hdinsight-architecture-hybrid.png

Links:
- azure.microsoft.com/en-us/services/hdinsight

Search as a Service (4)

Search as a service is a branch of software as a service (SaaS), focussed on enterprise search or site-specific web search. Enterprise search is the practice of making content from multiple enterprise-type sources, such as databases and intranets, searchable to a defined audience.

[AWS] CloudSearch
A managed service in the AWS Cloud that makes it simple and cost-effective to set up, manage, and scale a search solution for your website or application.
Amazon CloudSearch is a managed service in the AWS Cloud that makes it simple and cost-effective to set up, manage, and scale a search solution for your website or application. Amazon CloudSearch supports 34 languages and popular search features such as highlighting, autocomplete, and geospatial search.

Source: https://d1.awsstatic.com/whiteboard-graphics/products/CloudSearch/cloudsearcharchitecture.db1a0673525d850f37d2709a59b35c0d1b9f18df.png

Source: https://docs.aws.amazon.com/cloudsearch/latest/developerguide/images/cloudsearch-scaling-diagram.png

Links:
- aws.amazon.com/cloudsearch
[AWS] Elasticsearch Service
A fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost effectively at scale.
Amazon Elasticsearch Service is a fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost effectively at scale. You can build, monitor, and troubleshoot your applications using the tools you love, at the scale you need. The service provides support for open source Elasticsearch APIs, managed Kibana, integration with Logstash and other AWS services, and built-in alerting and SQL querying. Amazon Elasticsearch Service lets you pay only for what you use – there are no upfront costs or usage requirements. With Amazon Elasticsearch Service, you get the ELK stack you need, without the operational overhead.

Source: https://d1.awsstatic.com/re19/ESS_HIW.db99588614cbf44e2d62ef9c9c173ebfe41e2834.png

Links:
- https://aws.amazon.com/elasticsearch-service/
[Google Cloud] Search
With 100+ connectors, you can index your third-party content from dozens of enterprise sources.
Cloud Search delivers enterprise search through robust SDKs and ready-to-use APIs to help you scalably index vast amounts of data from any source. With 100+ connectors, you can index your third-party content from dozens of enterprise sources. This includes storage solutions like Amazon S3, Box, and Microsoft OneDrive; databases like Oracle, MySQL, and PostgreSQL; collaboration solutions like Atlassian Confluence, Jira, and Microsoft Sharepoint; documentation, ERP, sales, and support systems like Salesforce, SAP, and ServiceNow — just to name a few. Search and access your data from anywhere using our out-of-the box mobile and web apps, or integrate into custom apps or chatbots using our Query APIs. So whether your data is on-premises or in another cloud, you can stay on top of all the things you care about.

Source: https://developers.google.com/cloud-search/images/architecture-overview.png

Links:
- cloud.google.com/products/search
- developers.google.com/cloud-search/docs/guides
[Azure] Cognitive Search
A search-as-a-service cloud solution that gives developers APIs and tools for adding a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications.
Azure Cognitive Search (formerly known as "Azure Search") is a search-as-a-service cloud solution that gives developers APIs and tools for adding a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications. Your code or a tool invokes data ingestion (indexing) to create and load an index. Optionally, you can add cognitive skills to apply AI processes during indexing. Doing so can add new information and structures useful for search and other scenarios.

On the other side of your service, your application code issues query requests and handles responses. The search experience is defined in your client using functionality from Azure Cognitive Search, with query execution over a persisted index that you create, own, and store in your service.

Source: https://docs.microsoft.com/en-us/azure/search/media/search-what-is-azure-search/azure-search-diagram.svg

Links:
- en.wikipedia.org/wiki/Azure_Search

Data Lake as a Service (3)

The data lake stores raw data, in whatever form the data source provides. There is no assumptions about the schema of the data, each data source can use whatever schema it likes. It's up to the consumers of that data to make sense of that data for their own purposes.

Data Lake on AWS
A data lake solution that automatically configures the core AWS services necessary to easily tag, search, share, transform, analyze, and govern specific subsets of data across a company or with other external users.
AWS offers a data lake solution that automatically configures the core AWS services necessary to easily tag, search, share, transform, analyze, and govern specific subsets of data across a company or with other external users. The solution deploys a console that users can access to search and browse available datasets for their business needs. The solution also includes a federated template that allows you to launch a version of the solution that is ready to integrate with Microsoft Active Directory.

Source: https://d1.awsstatic.com/Solutions/Solutions%20Category%20Template%20Draft/Solution%20Architecture%20Diagrams/data-lake-architecture.e476d431350adc7a78a6161a0bb4ab95e6b09b6e.png

Links:
- aws.amazon.com/solutions/data-lake-solution
Cloud Storage as a Data Lake
A serverless, highly scalable, and cost-effective cloud data warehouse designed to help you make informed decisions quickly, so you can transform your business with ease.
Accelerate time-to-value with a fully managed and serverless cloud data warehouse that is easy to set up and manage and doesn’t require a database administrator. Jump-start data analysis cost effectively and uncover meaningful insights to stay competitive.

Source: https://cloud.google.com/solutions/images/data-lake-stages.svg

Source: https://cloud.google.com/solutions/images/data-lake-workflow.svg

Source: https://cloud.google.com/solutions/images/data-lake-bigquery.svg

Source: https://cloud.google.com/solutions/images/data-lake-batch-analytics.svg

Source: https://cloud.google.com/solutions/images/data-lake-real-time-analytics.svg

Source: https://cloud.google.com/solutions/images/data-lake-machine-learning.svg

Source: https://cloud.google.com/solutions/images/data-lake-pretrained-models.svg

Links:
- cloud.google.com/bigquery
[Azure] Data Lake
A no-limits data lake to power intelligent action.
Azure Data Lake includes all the capabilities required to make it easy for developers, data scientists, and analysts to store data of any size, shape, and speed, and do all types of processing and analytics across platforms and languages. It removes the complexities of ingesting and storing all of your data while making it faster to get up and running with batch, streaming, and interactive analytics. Azure Data Lake works with existing IT investments for identity, management, and security for simplified data management and governance. It also integrates seamlessly with operational stores and data warehouses so you can extend current data applications. We’ve drawn on the experience of working with enterprise customers and running some of the largest scale processing and analytics in the world for Microsoft businesses like Office 365, Xbox Live, Azure, Windows, Bing, and Skype. Azure Data Lake solves many of the productivity and scalability challenges that prevent you from maximizing the value of your data assets with a service that’s ready to meet your current and future business needs.

Source: https://azurecomcdn.azureedge.net/cvt-f5458f331e589f4df3ee1444da812347560174b9cfa47ae530103d627388294d/images/page/solutions/data-lake/data-lake-diagram.png

Links:

Public Cloud Encyclopedia: Data(base) Products

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links:

Links: