As seen, Snowflake has two ‘Scaling Policy’ choices. Understand them well.

Standard

Snowflake’s default scaling policy. For credit consumption, this strategy is more costly than ‘Economy’. As expected, it performs better. This strategy aims to reduce query queues.

If any of these are true, a cluster starts:

• Query queued

OR

• The system discovers an additional query beyond the capacity of the current clusters.

AND

• 20 seconds have passed since the previous cluster started.

If A or B is true, the first cluster starts immediately. A, B, or C must be true for each cluster. Assuming a maximum of 5 clusters, starting all 5 takes at least 80 seconds (one immediately, and 20 seconds between each cluster).

Every minute, the system checks Least Loaded Cluster load. The Least Loaded Cluster is shut down after 2–3 checks whether the system can disperse its load without spinning it up again.

Thus, if you have a spike load for 20 minutes and 1 to 5 clusters, all 5 will be up and operating within 80 seconds. After 1200 seconds, or the spike’s conclusion, clusters will decrease by 1 every 2-3 minutes.

Economy

As the name implies, this approach conserves credits and costs. Thus, queries may queue and take longer to finish. The algorithm spins off new clusters when there is enough workload to keep them active for at least 6 minutes. Thus, the system must queue enough inquiries before initiating a new cluster.

Like the ‘Standard’ Snowflake Scaling Policy, a cluster is stopped by monitoring the load on the least loaded cluster and seeing if the others can manage it. Instead of 2–3 checks, the system does 5–6 checks before shutting down a cluster here. Thus, cluster shutdown takes longer, thus consider this while choosing between ‘Standard’ and ‘Economy’ policies.

The ‘Economy’ Snowflake Scaling Policy saves money during brief surges when load is too low for a second cluster. However, brief to moderate delays and delayed query processing may occur.

Masking policies are automatically applied to columns depending on their associated tags. This feature provides consistent protection across tagged columns, avoiding the need for manual policy assignments. It works neatly with column data types and prefers explicitly assigned masking policies to tag-based ones.

1. JPMorgan Chase

JPMorgan Chase uses LLMs to process enormous volumes of financial documents for fraud detection, risk assessment, and compliance.

Challenge: The bank faced a difficult task that was prone to human error. It was extracting specific data from financial records, transaction logs, and legal documents.

Solution: To extract actionable insights such as interest rates, terms of payment, and other compliance issues, LLMs were used to scan documents. After that, these findings were prepared for reporting and analysis.

Outcome: By drastically cutting down on the amount of time needed for document analysis, JPMorgan Chase was better able to control risks and adhere to legal requirements.

2. Zurich Insurance

Challenge: It took a lot of manual work to process hundreds of insurance claims from handwritten notes and scanned documents.

Solution: Zurich Insurance used LLMs to extract information from unstructured claim forms, including dates, customer information, and claim amounts. For analysis, the retrieved data was automatically cleaned and organised.

Outcome: By cutting processing time by 80%, the automation accelerated claim approvals and raised customer satisfaction.

3. Unilever

Challenge: Unilever’s supply chain depended on data from multiple sources, often in inconsistent formats, including supplier records, invoices, and logistics reports.

Solution: From these documents, an LLM-driven system standardised them into a central database by extracting important information including supplier IDs, shipment dates, and quantities.

Outcome: Unilever was able to make proactive decisions and cut inventory expenditures by gaining real-time supply chain visibility.

4. Walmart

Challenge: To maximise product placements and pricing strategies, Walmart has to analyse enormous volumes of transaction data and customer feedback.

Solution: LLMs were implemented to process reviews, extract keywords, and classify consumer sentiments. Transaction logs were also compiled and cleaned to analyse trends.

Outcome: Walmart was able to improve consumer satisfaction, dynamically modify pricing, and identify underperforming products with the help of these insights.

5. Mayo Clinic

By incorporating LLMs into their data pipeline, they expedited the extraction of clinical insights from unstructured data such as medical records and physician notes.

Challenge: The Mayo Clinic needed to analyse millions of medical records to improve treatment techniques. However, 80% of their data was in unstructured text format, which made typical ETL operations time-consuming and error-prone.

Solution: Using GPT-based LLMs, the clinic automated the extraction of important information such as patient demographics, diagnosis codes, treatment plans, and outcomes. The LLMs converted raw text into structured data types that worked with their analytics systems.

Outcome: Automation reduced data preparation time by 70%, allowing researchers to analyse trends and improve treatment procedures and patient outcomes.

LLMs in ETL Pipelines

Every step of the ETL pipeline can benefit greatly from the use of LLMs:

1. Extract

Parsing scanned documents, photos, and unstructured text.
Example: Retrieving client information from insurance claims PDF forms.

2. Transform

Transforming unstructured data into a format suitable for analysis.
Example: Cleaning and classifying user reviews from e-commerce sites.

3. Load

Loading cleansed data into databases or visualisation programs through mapping.
Example: Loading organised medical data into a business intelligence dashboard.

Tools and Technologies

LLMs have been adopted by several tools and organisations to automate data preparation:

• Databricks

Within its Lakehouse platform, Databricks provides LLM-based solutions for organising and cleaning large amounts of data.

• Azure Synapse by Microsoft

Offers AI-powered tools for automating ETL procedures, such as LLM integrations.

• Snowflake

Simplifies data transformation and schema matching operations with AI and LLMs.

• Hugging Face and OpenAI

Offer APIs for NLP-based data preparation that can be included in custom ETL processes.

Snowflake provides several integrated functionalities to tackle these issues. Here is a method for their effective utilization: