Skip to main content

New top story on Hacker News: Launch HN: Baselit (YC W23) – Automatically Reduce Snowflake Costs

Launch HN: Baselit (YC W23) – Automatically Reduce Snowflake Costs
12 by sahil_singla | 9 comments on Hacker News.
Hey HN! We are Baselit ( https://baselit.ai/ ), a tool that automatically optimizes Snowflake costs. Here’s a demo video: https://www.youtube.com/watch?v=Ls6VRzBQ-pQ . Snowflake is one of the most widely used data warehouses today. It abstracts out the underlying compute infrastructure into “warehouses” for the user - compute units with t-shirt sizes (X-Small, Small, Medium etc.). In general, if you want to lower your data processing costs, the only thing you can do is to just process less data (i.e. query optimization). But Snowflake’s warehouse abstraction allows an extra dimension along which you can optimize - by minimizing the compute you need to process that same data (i.e. warehouse optimization). Baselit automates Snowflake warehouse optimization for you. While we were working on another idea last year (AI for SQL generation), users frequently shared with us how Snowflake costs had become a top concern, and cost optimization was now their business priority. Every few months, they would manually look for opportunities to cut down on costs (removing workloads or optimizing queries) - a time consuming process. We decided to build a solution that could automate cost optimization, and complement the manual effort of data teams. There are two key components of Baselit: 1. Automated agents that cut down on warehouse idle time. This happens in one of two ways: cache optimization (when to suspend a warehouse vs letting it run idle) and cluster optimization (optimal spin down of clusters). You can easily find out how much these agents can save you. Here’s a SQL query that you can run on your Snowflake, and it will calculate your savings - https://ift.tt/05fIkwM 2. Autoscaler that lets you create custom scaling policies for multi-cluster warehouses based on SLAs. Snowflake’s default policies (Economy and Standard) are not cost optimal in most cases, and they don’t give you any control. One use case for Autoscaler is that it helps you efficiently merge several warehouses into one multi-cluster warehouse - with a custom scaling policy that is optimal for a particular type of workload. In Autoscaler, you can set a parameter called “Allowed Queuing Time” that controls how fast a new cluster should spin up. For e.g. if you want to merge transformation workloads, you might want to set a higher queuing time. Baselit will slow down the cluster spin up, ensuring all clusters are running at a high utilization, and you’ll see a reduction in costs. We’ve built a bunch of other features that help in optimizing Snowflake costs: a dbt optimization feature that automatically picks the right warehouse size for dbt models through constant experimentation, a “cost lineage”, spend views by teams/roles/users, and automatic recommendations from scanning Snowflake metadata. Due to the nature of our product (access to Snowflake metadata required), we haven’t made Baselit self-serve yet. We invite you to run our savings query ( https://ift.tt/05fIkwM ) and find out your potential savings. And if you’d like to know more about any of our features and get a live demo, you can book one at this link - https://ift.tt/g5X48q9 We’d love to read your feedback and ideas on Snowflake optimization!

Comments

Popular posts from this blog

New top story on Hacker News: Ask HN: Is the EULA on my new $30k RED cinema camera legal?

Ask HN: Is the EULA on my new $30k RED cinema camera legal? 70 by red_throwaway | 77 comments on Hacker News. TLDR: I bought a $30K professional cinema camera that doesn't work unless I sign away my rights to privacy and possibly the video content I make with it ( at least it seems ) Over the past few years my photography business has seen a surge in demand for ultra high quality video production work. In an effort to meet this demand, I picked up one of RED Digital Cinema's newest pro camera bodies, the RED V-RAPTOR. Considering this camera is used by professional filmmakers to create films destined for cinemas, it's not surprising that it came with a $30k price tag. After unboxing and assembling it, I power the camera on and the first thing I see is a wall of legal text on the embedded LCD. Turns out it's a "Software License Agreement" that I'm required to consent to using the on-camera menu buttons before any of the camera's functionality becomes...

New top story on Hacker News: Launch HN: Spoken (YC S21) – Better furniture shopping

Launch HN: Spoken (YC S21) – Better furniture shopping 14 by lgabraham | 3 comments on Hacker News. Hi HN, we’re Dane and Geoff, the founders of Spoken ( https://www.spoken.io/ ). We make it easy to find the lowest price for any furniture item across all big stores. Buying physical things on the internet is hard. You have to quality-check a product without touching it, double-check dimensions for where the thing will go, and evaluate a seller's credibility, often with little data. But buying furniture online is a special case of hard, because the market is deliberately deceptive. Furniture sellers actively prevent consumers from easily finding the same item at other stores, or under other names, because this allows them to charge more. The sellers get to name the products and they name them in confusing ways to facilitate price discriminaton. For example, this table at Wayfair ( https://ift.tt/RF071cQ... ) can also be found at Appliances Connection under a different name for rou...