Globally and Locally Consistent Image Completion

  • Published on
    24-Jan-2018

  • View
    587

  • Download
    9

Embed Size (px)

Transcript

  • Globally and LocallyConsistent Image Completion

    SATOSHI IIZUKA , EDGAR SIMO-SERRA, HIROSHI ISHIKAWA WasedaUniversity

    SIGGRAPH 2017 accepted

    http://hi.cs.waseda.ac.jp/~iizuka/projects/completion/data/completion_sig2017.pdf

    http://hi.cs.waseda.ac.jp/~iizuka/projects/completion/data/completion_sig2017.pdf

  • SIGGRAPH

    First Author:() DL

    - AI - 2

    HPhttp://hi.cs.waseda.ac.jp/~iizuka/

  • (inpainting)

    GAN (Generative Adversarial Networks)

    2Discriminator

    - AI - 3

  • - AI - 4

    Context encoder

    Image size

    Local Consistency

    Semantics

    Novel objects

    Image size

    Local Consistency

    Semantics

    Novel objects

    Context encoder

  • Barnes et al. 2009; Darabi et al. 2012; Huang et al. 2014; Simakov et al. 2008;

    Wexler et al. 2007

    - AI - 5

  • Barnes et al. 2009; Darabi et al. 2012; Huang et al. 2014; Simakov et al. 2008;

    Wexler et al. 2007

    - AI - 6

  • ex)

    - AI - 7

  • Context Encoders Context Encoders: Feature Learning by Inpainting

    Pathak et al. 2016, CVPR2016

    AutoEncoderGAN

    - AI - 8

  • Context Encoders

    AutoEncoder

    - AI - 9

    AlexNetAutoEncoder

    :L2L1

  • Context Encoders

    GAN

    - AI - 10

    AlexNetAutoEncoder

    Generator

    Discriminator

    GAN

  • Context Encoders L2 Loss

    = ( ( 1 )) 2 (1)

    GAN

    Generator Discriminator

    min

    max

    log + [log 1 ] (2)

    Adversarial Loss

    = max

    log + log 1 ( ( 1 )) (3)

    = + (4)

    - AI - 11

  • Context Encoders

    - AI - 12

  • Globally and Locally Consistent Image Completion (GLCIC)

    Context Encoder

    Context Encoder

    - AI - 13

  • (dilated convolution layers [Yu and Koltun 2016])

    CC

    x, C

    , C

    , C x C

    ()

    ( = 1 > 1)

    , = + =

    =

    +,

    + +,+ (5)

    =

    1

    2

    =1

    2

    - AI - 14

  • 1. (AutoEncoder)

    2.

    - AI - 15

  • (Hole)

    307x30799x99

    - AI - 16

  • 128 x

    128

    - AI - 17

    Concatenation 10242048

    1

    sigmoid [0, 1]

  • C

    min

    max

    [ , + log , + log(1 , , ) ] (6)

    Context Encoders

    Adversarial Loss

    - AI - 18

  • 11t

    t < TC L2 Loss

    TC < t < TC + TD

    t > TC + TD

    t = Ttrain

    TC = 90,000TD = 10,000Ttrain = 500,000: 96

    : AdaDelta

    - AI - 19

  • Places2 (http://places2.csail.mit.edu/)

    4001000

    8097967800

    [256, 384]

    [96, 128]99x99

    fast marching method [Telea 2004]

    - AI - 20

    http://places2.csail.mit.edu/)

  • - AI - 21

    ex)

  • - AI - 22

  • - AI - 23

  • - AI - 24

  • - AI - 25

  • - AI - 26

  • - AI - 27

    ImageNet: 1

    Places2: 8

    Places2

  • - AI - 28

  • - AI - 29

    FineTuning CelebFaces Attributes Dataset (CelebA)

    lCMP Facade Dataset

  • - AI - 30

  • - AI - 31

    FineTuning

    FineTuning

  • - AI - 32

  • - AI - 33

  • CelebA ()

    10

    - AI - 34

  • 77%

    - AI - 35