Reading drill

  • View
    151

  • Download
    0

Embed Size (px)

Transcript

  • DrillKai Sasaki(@Lewuathe)

  • (@Lewuathe)

    Hadoop, Storm

    Spark MLLib

    Big DataDeep Learning

  • Apache Drill MapRSQL

    GoogleDremel

    Impala (http://impala.io/)

    Phoenix (http://phoenix.apache.org/)

    Presto (https://prestodb.io/)

  • Drill

    Drill

    Drill

  • Dremel Google

    BigQuery

    Dremel: Interactive Analysis of Web-Scale Datasets"

  • Dremel

    Columnar Storage Format

    NodeAggregation

  • Columnar Storage Format

    http://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/36632.pdf

  • Columnar Storage Format

    http://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/36632.pdf

  • Columnar Storage Format

    http://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/36632.pdf

    instantiate

  • Columnar Storage Format

    http://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/36632.pdf

    CSF

    r: Repetition Level d: Definition Lebels

    Definition Level

  • Columnar Storage Format

    http://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/36632.pdf

    CSF

    r: Repetition Level d: Definition Lebels

    Loss Less

    Definition Level

  • Agility IT

    Flexibility

    Familiarity ANSI SQLJDBC/ODBCBI

  • Agility

    Embedded Mode service

    Distributed Mode ProductionDrillbitZooKeeper

  • AgilityDistributed ModeCookbook

    https://supermarket.chef.io/cookbooks/drill

  • Flexibility Data SourcePlugin

  • Flexibility Data SourcePlugin

  • Familiarity,'%&'3+..7-.1&$. !.10'10/$+05(/2 #10'10!(/2#.10'10/$+05(/2 #10'10!(/2#')4#.10'10"6($5*(3,410#.10'10')4#.10'10"6($5*(3,410#.10'10.10'100$/(.10'100$/(!810'10!(/2810'10!(/28888

    3164(.(&5('4(&10'4

    JSONjoin

  • SQLLine

    RDBMS

    OptiqJDBCSQL"

    DrillbitDrill

  • SQLLine

    1.

    Optiq

    DrillClient

    Drillbit Drillbit Drillbit Drillbit Drillbit

  • SQLLine

    1.

    Optiq

    DrillClient

    Drillbit Drillbit Drillbit Drillbit Drillbit

    2. JDBC

  • SQLLine

    1.

    Optiq

    DrillClient

    Drillbit Drillbit Drillbit Drillbit Drillbit

    2. JDBC

    3. Logical Plan

  • SQLLine

    1.

    Optiq

    DrillClient

    Drillbit Drillbit Drillbit Drillbit Drillbit

    2. JDBC

    3. Logical Plan

    4. ProtocolBufferPlan

  • SQLLine

    1.

    Optiq

    DrillClient

    Drillbit Drillbit Drillbit Drillbit DrillbitForeman

    2. JDBC

    3. Logical Plan

    4. ProtocolBufferPlan

    5. Foreman

  • SQLLine

    1.

    Optiq

    DrillClient

    Drillbit Drillbit Drillbit Drillbit DrillbitForeman

    2. JDBC

    3. Logical Plan

    4. ProtocolBufferPlan

    5. Foreman

    6. Planfragment

  • JDBC, ODBCSQL

    SPOF

    Fragment

  • ContributionSQL Functions Add, AVG, ConvertSQLexec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl

    New File FormatXML, Apache Avro, Thrift"

    New Data Source Apache Solr, Elastic Search, MySQL, HTTP URL

    New Query LanguagePig Latin, HiveQL

  • DrillSelf-DrivenSQL

    SQL