Семантическое ядро рунета

  • Published on
    16-Apr-2017

  • View
    114

  • Download
    0

Embed Size (px)

Transcript

2011

XII
CEE-SECR / 28 - 29 , Click to edit Master title style

Click to edit Master title style

Click to edit Master text stylesSecond levelThird levelFourth level

Fifth level

28.10.2016

21.6.16

21.6.16

C:\Users\filippov\Desktop\logo.png

ontent-based Amazon Kinesis/Lucene

1-

?

/

,

,

/

Best-sellers

-

Mining of Massive Datasets, 9.1.2: Leskovec, Rajaraman, Ullman (Stanford University)

/

/, (User-User)

(Item-Item):

/

( )

// ()

Apache Spark MLlib (als), Apache Mahout (Taste) + ,

/

Content-based

.

Toyota, , Toyota

: Sphinx, Lucene (Solr)

: . , .

/

(, , )

/

, , ,

/

?

?

?

?

...

/

- RabbitMQ

http://www.rabbitmq.com

AMQP

Erlang

/

- Apache Kafka

http://kafka.apache.org/

LinkedIn

!

Scala

/

- Apache Storm

http://storm.apache.org

Task parallel

, workflow

Clojure/JVM

/

- Pinba

http://pinba.org

. MySQL PHP

,

Badoo.com

/

Apache Hadoop

:

- (MapReduce)

- (HDFS)

- SQL- (Hive)

/

Apache Spark

!

/

MapReduce

Mining of Massive Datasets: Leskovec, Rajaraman, Ullman (Stanford University)

/

Apache Spark

/

Online , !

SQL ;-)

Mining of Massive Datasets: Leskovec, Rajaraman, Ullman (Stanford University)

/

SQL MapReduce: Hive, Pig, Spark SQL

SQL MPP (massive parallel processing):

Impala, Presto, Amazon RedShift, Vertica

NoSQL: Cassandra, Hbase, Amazon DynamoDB

: MySQL, MS SQL, Oracle,

/

Lambda-

/

BigData . .

ID

ID


/

BigData . .

~1000 /

bitrix.info

,

Batch

On-line

analytics.bitrix.info

/

Amazon DynamoDB

nginx+Lua

Amazon Kinesis

BigData . .

~1000 /

bitrix.info

workers cluster

worker (PHP)

~100 /

worker (PHP)

worker (PHP)

worker (PHP)

worker (PHP)

worker (PHP)

worker (PHP)

worker (PHP)

/

Amazon DynamoDB

BigData . , , .

Apache Spark

(spot)

Amazon S3

(spot)

(spot)

(spot)

Apache Tomcat

Apache Mahout

analytics.bitrix.info

/

Apache Lucene

Doug Cutting: Nutch, Hadoop (Yahoo!) Cloudera

Lucene: Solr, ElasticSearch

Lucene:

/

Apache Lucene: +/-

(-)

(-)

(-)

(-) 100%

(+) ()

(+) API

(+)

(+) Thread-safety

/

Redis

:

(word2vec, glove, ...)

/

Amazon Kinesis

Java indexing workers (16)

~1000 /

content-based

Index (disk)

Index (disk)

Redis (profiles)

Servlet

/

, java/lucene

Amazon Kinesis

,

Servlet

/

: -

: , 100

: , h1

: , ,

: ,

/

/


! ?

@AlexSerbulserbul@1c-bitrix.ru