Merge pull request #271 from FluxML/doc

Update docs

Author: yuehhua

README.md

@@ -28,32 +28,35 @@ Suggestions, issues and pull requsts are welcome.
 * Variable graph inputs are supported. You use it when diverse graph structures are prepared as inputs to the same model.
 * Integrate GNN benchmark datasets (WIP)
 
-## Featured Graphs
+### Featured Graphs
 
 GeometricFlux handles graph data (the topology plus node/vertex/graph features)
-thanks to the type `FeaturedGraph`.
+thanks to `FeaturedGraph` type.
 
-A `FeaturedGraph` can be constructed out of 
+A `FeaturedGraph` can be constructed from various graph structures, including
 adjacency matrices, adjacency lists, Graphs' types...
 
 ```julia
 fg = FeaturedGraph(adj_list)   
 ```
-## Graph convolutional layers
+
+### Graph convolutional layers
 
 Construct a GCN layer:
 
 ```julia
-GCNConv([fg,] input_dim => output_dim, relu)
+GCNConv(input_dim => output_dim, relu)
 ```
 
 ## Use it as you use Flux
 
 ```julia
-model = Chain(GCNConv(fg, 1024 => 512, relu),
-              Dropout(0.5),
-              GCNConv(fg, 512 => 128),
-              Dense(128, 10))
+model = Chain(
+    WithGraph(fg, GCNConv(fg, 1024 => 512, relu)),
+    Dropout(0.5),
+    WithGraph(fg, GCNConv(fg, 512 => 128)),
+    Dense(128, 10)
+)
 ## Loss
 loss(x, y) = logitcrossentropy(model(x), y)
 accuracy(x, y) = mean(onecold(model(x)) .== onecold(y))

docs/bibliography.bib

@@ -15,3 +15,9 @@ @inproceedings{node2vec2016
     location = {San Francisco, California, USA},
     series = {KDD '16}
 }
+
+@misc{google_word2vec,
+    title = {Google code archive - long-term storage for google code project hosting.},
+    url = {https://code.google.com/archive/p/word2vec/},
+    publisher = {Google}
+}

docs/make.jl

@@ -13,27 +13,32 @@ makedocs(
       analytics = "G-M61P0B2Y8E",
     ),
     clean = false,
-    modules = [GeometricFlux],
+    modules = [GeometricFlux,GraphSignals],
     pages = ["Home" => "index.md",
-             "Get started" => "start.md",
-             "Basics" =>
-               ["Graph convolutions" => "basics/conv.md",
-                "Building layers" => "basics/layers.md",
-                "Graph passing" => "basics/passgraph.md"],
-             "Cooperate with Flux layers" => "cooperate.md",
-             "Tutorials" =>
-                [
-                  "Semi-supervised learning with GCN" => "tutorials/semisupervised_gcn.md",
+             "Introduction" => "introduction.md",
+             "Basics" => [
+                 "Graph Convolutions" => "basics/conv.md",
+                 "Graph Passing" => "basics/passgraph.md",
+                 "Building Layers" => "basics/layers.md",
+                 "Subgraph" => "basics/subgraph.md",
+                 "Batch Learning" => "basics/batch.md",
+                ],
+             "Cooperate with Flux Layers" => "cooperate.md",
+             "Tutorials" => [
+                  "Semi-Supervised Learning with GCN" => "tutorials/semisupervised_gcn.md",
                   "GCN with Fixed Graph" => "tutorials/gcn_fixed_graph.md",
                 ],
-             "Abstractions" =>
-               ["Message passing scheme" => "abstractions/msgpass.md",
-                "Graph network block" => "abstractions/gn.md"],
-             "Manual" =>
-               ["Convolutional Layers" => "manual/conv.md",
-                "Pooling Layers" => "manual/pool.md",
-                "Models" => "manual/models.md",
-                "Linear Algebra" => "manual/linalg.md"],
+             "Abstractions" => [
+               "Message passing scheme" => "abstractions/msgpass.md",
+               "Graph network block" => "abstractions/gn.md"],
+             "Manual" => [
+               "FeaturedGraph" => "manual/featuredgraph.md",
+               "Convolutional Layers" => "manual/conv.md",
+               "Pooling Layers" => "manual/pool.md",
+               "Embeddings" => "manual/embedding.md",
+               "Models" => "manual/models.md",
+               "Linear Algebra" => "manual/linalg.md"
+               ],
              "References" => "references.md",
     ]
 )

docs/src/abstractions/msgpass.md

@@ -11,10 +11,18 @@ update(mp, m, x) = m
 
 `mp` denotes a message passing layer. `message` accepts node state `x_i` for node `i` and its neighbor state `x_j` for node `j`, as well as corresponding edge state `e_ij` for edge `(i,j)`. The default message function gives all the neighbor state `x_j` for neighbor of node `i`. `update` takes aggregated message `m` and current node state `x`, and then outputs `m`.
 
+```@docs
+GeometricFlux.MessagePassing
+```
+
 ## Message function
 
 A message function accepts feature vector representing node state `x_i`, feature vectors for neighbor state `x_j` and corresponding edge state `e_ij`. A vector is expected to output from `message` for message. User can override `message` for customized message passing layer to provide desired behavior.
 
+```@docs
+GeometricFlux.message
+```
+
 ## Aggregate messages
 
 Messages from message function are aggregated by an aggregate function. An aggregated message is passed to update function for node-level computation. An aggregate function is given by the following:
@@ -38,3 +46,7 @@ The following `aggr` are available aggregate functions:
 ## Update function
 
 An update function takes aggregated message `m` and current node state `x` as arguments. An output vector is expected to be the new node state for next layer. User can override `update` for customized message passing layer to provide desired behavior.
+
+```@docs
+GeometricFlux.update
+```

docs/src/assets/bypass_graph.svg

@@ -0,0 +1,23 @@
+# Batch Learning
+
+## Batch Learning for Variable Graph Strategy
+
+Batch learning for variable graph strategy can be prepared as follows:
+
+```julia
+train_data = [(FeaturedGraph(g, nf=train_X), train_y) for _ in 1:N]
+train_batch = Flux.batch(train_data)
+```
+
+It batches up `FeaturedGraph` objects into specified mini-batch. A batch is passed to a GNN model and trained/inferred one by one. It is hard for `FeaturedGraph` objects to train or infer in real batch for GPU.
+
+## Batch Learning for Static Graph Strategy
+
+A efficient batch learning should use static graph strategy. Batch learning for static graph strategy can be prepared as follows:
+
+```julia
+train_data = (repeat(train_X, outer=(1,1,N)), repeat(train_y, outer=(1,1,N)))
+train_loader = DataLoader(train_data, batchsize=batch_size, shuffle=true)
+```
+
+An efficient batch learning should feed array to a GNN model. In the example, the mini-batch dimension is the third dimension for `train_X` array. The `train_X` array is split by `DataLoader` into mini-batches and feed a mini-batch to GNN model at a time. This strategy leverages the advantage of GPU training by accelerating training GNN model in a real batch learning.

docs/src/assets/geometry.svg

@@ -1,3 +1,3 @@
-# Graph convolutions
+# Graph Convolutions
 
-Graph convolution can be classified into spectral-based graph convolution and spatial-based graph convolution. Spectral-based graph convolution, such as `GCNConv` and `ChebConv`, performs operation on features of *whole* graph at one time. Spatial-based graph convolution, such as `GraphConv` and `GATConv`, performs operation on features of *local* graph instead. Message-passing scheme is an abstraction for spatial-based graph convolutional layers. Any spatial-based graph convolutional layer can be implemented under the framework of message-passing scheme.
+Graph convolution can be classified into *spectral-based graph convolution* and *spatial-based graph convolution*. Spectral-based graph convolution, such as `GCNConv` and `ChebConv`, performs operation on features of *whole* graph at one time. Spatial-based graph convolution, such as `GraphConv` and `GATConv`, performs operation on features of *local* subgraph instead. Message-passing scheme is an abstraction for spatial-based graph convolutional layers. Any spatial-based graph convolutional layer can be implemented under the framework of message-passing scheme.

docs/src/assets/graph signals.svg

@@ -1,26 +1,85 @@
-# Building graph neural networks
+# Building Graph Neural Networks
 
 Building GNN is as simple as building neural network in Flux. The syntax here is the same as Flux. `Chain` is used to stack layers into a GNN. A simple example is shown here:
 
 ```julia
-model = Chain(GCNConv(adj_mat, feat=>h1),
-              GCNConv(adj_mat, h1=>h2, relu))
+model = Chain(
+    GCNConv(feat=>h1),
+    GCNConv(h1=>h2, relu),
+)
 ```
 
-In the example above, The first argument `adj_mat` is the representation of a graph in form of adjacency matrix. The feature dimension in first layer is mapped from `feat` to `h1`. In second layer, `h1` is then mapped to `h2`. Default activation function is given as identity if it is not specified by users.
+In the example above, the feature dimension in first layer is mapped from `feat` to `h1`. In second layer, `h1` is then mapped to `h2`. Default activation function is given as `identity` if it is not specified by users.
 
-The initialization function `GCNConv(...)` constructs a `GCNConv` layer. For most of the layer types in GeometricFlux, a layer can be initialized in at least two ways:
+The initialization function `GCNConv(...)` constructs a `GCNConv` layer. For most of the layer types in GeometricFlux, a layer can be initialized in two ways:
 
-* Initializing *with* a predefined adjacency matrix or `FeaturedGraph`, followed by the other parameters. For most of the layer types, this is for datasets where each input has the same graph structure.
-* Initializing *without* an initial graph argument, only supplying the relevant parameters. This allows the layer to accept different graph structures.
+* GNN layer without graph: initializing *without* a predefined graph topology. This allows the layer to accept different graph topology.
+* GNN layer with static graph: initializing *with* a predefined graph topology, e.g. graph wrapped in `FeaturedGraph`. This strategy is suitable for datasets where each input requires the same graph structure and it has better performance than variable graph strategy.
 
-# Applying layers
+The example above demonstrate the variable graph strategy. The equivalent GNN architecture but with static graph strategy is shown as following:
+
+```julia
+model = Chain(
+    WithGraph(fg, GCNConv(feat=>h1)),
+    WithGraph(fg, GCNConv(h1=>h2, relu)),
+)
+```
+
+```@docs
+GeometricFlux.WithGraph
+```
+
+## Applying Layers
 
 When using GNN layers, the general guidelines are:
 
-* If you pass in a ``n \times d`` matrix of node features, and the layer maps node features ``\mathbb{R}^d \rightarrow \mathbb{R}^k`` then the output will be in matrix with dimensions ``n \times k``. The same ostensibly goes for edge features but as of now no layer type supports outputting new edge features.
-* If you pass in a `FeaturedGraph`, the output will be also be a `FeaturedGraph` with modified node (and/or edge) features. Add `node_feature` as the following entry in the Flux chain (or simply call `node_feature()` on the output) if you wish to subsequently convert them to matrix form.
+* With static graph strategy: you should pass in a ``d \times n \times batch`` matrix for node features, and the layer maps node features ``\mathbb{R}^d \rightarrow \mathbb{R}^k`` then the output will be in matrix with dimensions ``k \times n \times batch``. The same ostensibly goes for edge features but as of now no layer type supports outputting new edge features.
+* With variable graph strategy: you should pass in a `FeaturedGraph`, the output will be also be a `FeaturedGraph` with modified node (and/or edge) features. Add `node_feature` as the following entry in the Flux chain (or simply call `node_feature()` on the output) if you wish to subsequently convert them to matrix form.
+
+## Define Your Own GNN Layer
+
+Customizing your own GNN layers are the same as defining a layer in Flux. You may want to check [Flux documentation](https://fluxml.ai/Flux.jl/stable/models/basics/#Building-Layers-1) first.
+
+To define a customized GNN layer, for example, we take a simple `GCNConv` layer as example here.
+
+```julia
+struct GCNConv <: AbstractGraphLayer
+    weight
+    bias
+    σ
+end
+
+@functor GCNConv
+```
+
+We first should define a `GCNConv` type and let it be the subtype of `AbstractGraphLayer`. In this type, it holds parameters that a layer operate on. Don't forget to add `@functor` macro to `GCNConv` type.
+
+```julia
+(l::GCNConv)(Ã::AbstractMatrix, x::AbstractMatrix) = l.σ.(l.weight * x * Ã .+ l.bias)
+```
 
-## Create custom layers
+Then, we can define the operation for `GCNConv` layer.
+
+```julia
+function (l::GCNConv)(fg::AbstractFeaturedGraph)
+    nf = node_feature(fg)
+    Ã = Zygote.ignore() do
+        GraphSignals.normalized_adjacency_matrix(fg, eltype(nf); selfloop=true)
+    end
+    return ConcreteFeaturedGraph(fg, nf = l(Ã, nf))
+end
+```
 
-Customizing your own GNN layers are the same as customizing layers in Flux. You may want to reference [Flux documentation](https://fluxml.ai/Flux.jl/stable/models/basics/#Building-Layers-1).
+Here comes to the GNN-specific behaviors. A GNN layer should accept object of subtype of `AbstractFeaturedGraph` to support variable graph strategy. A variable graph strategy should fetch node/edge/global features from `fg` and transform graph in `fg` into required form for layer operation, e.g. `GCNConv` layer needs a normalized adjacency matrix with self loop. Then, normalized adjacency matrix `Ã` and node features `nf` are pass through `GCNConv` layer `l(Ã, nf)` to give a new node feature. Finally, a `ConcreteFeaturedGraph` wrap graph in `fg` and new node features into a new object of subtype of `AbstractFeaturedGraph`.
+
+```julia
+layer = GCNConv(10=>5, relu)
+new_fg = layer(fg)
+gradient(() -> sum(node_feature(layer(fg))), Flux.params(layer))
+```
+
+Now we complete a simple version of `GCNConv` layer. One can test the forward pass and gradient if they work properly.
+
+```@docs
+GeometricFlux.AbstractGraphLayer
+```