RSS

Tag Archives: architecture

Data Warehouse Solutions in Azure

Date Warehousing Solutions at a Glance

With today’s big data requirements where data could be structured, unstructured, batch, stream and come in many other forms and size, traditional data warehouse is not going to cut it.

Typically, there are 4 types of data stage:

  • Ingest
  • Store
  • Processing
  • Consuming

Different technology is required at different stage. This also depends heavily on size and form of data and the 4 Vs: Volume, Variety, Velocity, Veracity.

Consideration for the solutions sometime also depends on:

  • Ease of management
  • Team skill sets
  • Language
  • Cost
  • Specification / requirements
  • Integration with existing / others system.

Azure Services

Azure offers many services for data warehouse solutions. Traditionally, data warehouse has been ETL process + relational database storage like SQL Data Warehouse. Today, that may not always be the case.

Some of Azure services for data warehousing:

  • Azure HDInsight
    Azure offers various cluster types that comes with HDInsight, fully managed by Microsoft, but still require management from users. Also supports Data Lake Storage. More about HDInsight. HDInsight sits on “Processing” data stage.
  • Azure Databricks
    Its support for machine learning, AI, analytics and stream / graph processing makes it a go-to solution for data processing. It’s also fully integrated with Power BI and other source / destination tools. Notebooks in Databricks allows collaboration between data engineers, data scientist and business users. Compare to HDInsight.
  • Azure Data Factory
    The “Ingest” part of data stage. Its function is to bring data in and move them around different system. Azure Data Factory supports different pipelines across Azure services to connect the data and even on-premise data. Azure Data Factory can be used to control the flow of data.
  • Azure SQL Data Warehouse
    Typically the end destination of data and to be consumed by business users. SQL DW is platform as a service, require less management from users and great for team who already familiar with TSQL and SSMS (SQL Management Studio). You can also scale it dynamically, pause / resume the compute. SQL DW uses internal storage to store data and include the compute component. SQL Data Warehouse sits on “Consuming” stage.
  • Database services (RDBMS, Cosmos, etc)
    SQL database, or other relational database system, Cosmos are part of the storage solutions offered in Azure Services. This is typically more expensive than Azure Storage, but also offer other features. Database services are part of “Storage” stage.
  • Azure Data Lake Storage
    Build on top of Azure Storage, ADLS offers unlimited storage and file system based on HDFS, allowing optimization for analytics purpose, like Hadoop or HDInsight. ADLS is part of “Storage” stage.
  • Azure Data Lake Analytics
    ADLA is a high-level abstraction of HDInsight. Users will not need to worry about scaling and management of the clusters at all, it’s an instant scale per job. However, this also comes with some limitations. ADLA support USQL, a SQL-like language that allows custom user defined function in C#. The tooling is also what developers are already familiar with, Visual Studio.
  • Azure Storage
  • Azure Analysis Services
  • Power BI

Which one to use?

There’s no right or wrong answer. The right solution depends on many others things, technical and non-technical as well as the considerations mentioned above.

Simon Lidberg and Benjamin Wright Jones have a really good presentation around this topic. See the link at reference for their full talk. But, basically, the flowchart to make decision looks like this:

data-warehouse-solutions-in-azure

Reference

https://myignite.techcommunity.microsoft.com/sessions/66581

Advertisements
 
Leave a comment

Posted by on January 20, 2019 in General

 

Tags: , , , , , , , , , , , , , , , , , ,

What is Event Sourcing?

Event Sourcing

Every application has a state, representing current snapshot at the moment. Event sourcing is the idea of persisting all of these states as it changes from one to another.

Imagine a workflow for an invoice goes something like: invoice is created which set the status to pending. Pending invoice then gets approved which sets it approved. Approved invoice can be denied or collected. When invoice is denied, it goes back to pending, but when it’s collected, it becomes closed. The state of an invoice is where it’s at a particular time. However, when event sourcing pattern is implemented, we will be able to see a historical timeline of an invoice, from when it’s created to the current state.

What this allows us to do is, we are able to:

  • Complete rebuild: getting rid of the current state and use events to rebuild the invoice to its current state.
  • Temporal query: we can tell what state an invoice is at particular point in time.
  • Event replay: when the past event is incorrect, we can make the change and compute it from the beginning, thus, replaying the whole series of events.

Some applications of event sourcing are:

  • Source control. Every source control uses time series where commits made change the code. By storing events for every commit, we are able to state of the code at particular point in time.
  • Redo / undo. Applications that utilize undo / redo, such as word processor, will benefit from event sourcing pattern.
  • Document history. Word processor application will also benefit from event sourcing by storing event every time changes are made.

I hope this simple explanation of event sourcing helps you to understand what it is and how we can use it. There’s definitely a lot more to it that I don’t cover here, but there are lots of resources online you can learn from.

Reference

https://martinfowler.com/eaaDev/EventSourcing.html

 
Leave a comment

Posted by on November 29, 2018 in General, References

 

Tags: , , ,

Event Sourcing Design Pattern

What is Event Sourcing?

Is a architecture design pattern that uses append-only store to record full series of events that happen on specific data. The store also acts as publisher to all subscribers whenever an event is recorded.

Characteristics

  • Append-only store. Some implementation may be different.
  • Event store as system of record.
  • Materialized object is constructed from events.
  • Provide audit trails.
  • Fast performance.

Considerations

  • Undo is adding another event. Perform undo can only be done by adding compensating event since update is not allowed in Event Store
  • Migration to new format. All existing events must be updated.
  • Consistency. Multi-threaded applications may store events in the same event store. The consistency and order must be maintained.
  • Lack of transactions. Event store by its nature is not transactional and it’s eventual consistency. It’s possible to have event to reduce the inventory count while customer is placing order for the same item.
  • Performance. Replaying large amount of events can slow down the result for materialized view.

Reference
Microsoft Docs

 
Leave a comment

Posted by on May 25, 2018 in General

 

Tags: , ,

 
%d bloggers like this: