Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Set up a Hive Metastore

Step-by-step guide to setting up a Hive Metastore with Terraform if you don't already have one.

In case you don't have a Hive Metastore, we have provided this step-by-step guide to create one for you. But before we jump into that, it's important to understand what Hive Metastore brings to the table.

Hive Metastore is a component that stores all the structure information(metadata) of objects like tables and partitions in the warehouse including column and column type information. To know more about Hive Metastore and its architecture, please refer to this article https://cwiki.apache.org/confluence/display/hive/design#Design-Metastore

Now let's go ahead and set up the Hive Metastore.

We have tried to simplify this process as much as possible but if you still face any issues you can email us at [email protected].

  1. We have created terraform scripts so that you can launch Hive Metastore in your own environment.

    1. For AWS

  2. You can also build your own Metastore on AWS.

  3. Once the Hive Metastore is setup, We need to make sure that we,

    1. Create a Schema SQL file according to your S3 data

    2. Connect Presto with Hive Metastore

    3. Run Schema SQL in presto to populate Schema and statistics in Hive Metastore using Presto

Create a Schema SQL file according to your S3 data

The customer needs to prepare the SQL file with respect to the data, they want to query. We have provided the sample SQL files for you to go ahead and try it out.

Connecting Presto with Hive Metastore

Currently, Presto, Spark & many other engines can add schemas to Hive Metastore. We prefer Presto for the same. In case you don't have Presto with you, we have already created terraform scripts so that you can launch Presto in your own environment.

  1. For AWS

Run Schema SQL in presto to populate Schema and statistics in Hive Metastore using Presto

According to the Schema SQL file, the following operations will be performed.

  1. Create Database

  2. Create Table

  3. Repair Table ( Only If the table is partitioned)

  4. Analyze Table - Collect stats about a table so that it can be used for Cost Based optimization

For AWS: Login to the presto machine on EMR and run the following command:

Last updated

Was this helpful?