Compare commits

...

62 Commits

Author SHA1 Message Date
Alex Ling
3abd2924d0 Merge pull request #156 from hkalexling/dev
v0.20.0
2021-02-02 12:16:05 +08:00
Alex Ling
21233df754 Fix group filter on the download page 2021-02-01 11:37:00 +00:00
Alex Ling
ca8e9a164e Fix the /api page error when using base URL 2021-01-30 10:54:21 +00:00
Alex Ling
cd268af9dd Fix tags.css base URL 2021-01-30 07:39:54 +00:00
Alex Ling
135fa9fde6 Update sample config in README [skip ci] 2021-01-29 14:42:58 +00:00
Alex Ling
77333aaafd Bump version to 0.20.0 2021-01-29 10:27:31 +00:00
Alex Ling
1fad530331 Fix admin page theme setting syncing (#155) 2021-01-29 08:40:50 +00:00
Alex Ling
a1bd87098c Escape single quotes in migration 8 2021-01-28 12:41:51 +00:00
Alex Ling
a389fa7178 Allow delete all missing items (#151) 2021-01-28 09:55:41 +00:00
Alex Ling
b5db508005 Fix relative path mismatch (#151) 2021-01-28 04:04:42 +00:00
Alex Ling
30178c42ef Merge branch 'master' into dev 2021-01-27 09:47:49 +00:00
Alex Ling
b712db9e8f Merge pull request #154 from hkalexling/hkalexling-patch-1
Update autoapproval.yml
2021-01-27 16:28:05 +08:00
Alex Ling
dd9c75d1c9 Update autoapproval.yml 2021-01-27 16:17:33 +08:00
Alex Ling
2d150c3bf2 Create autoapproval.yml 2021-01-27 16:14:47 +08:00
Alex Ling
40f74ea375 Merge pull request #153 from hkalexling/hotfix/reader-bg
Fix incorrect background color on reader page
2021-01-27 15:19:04 +08:00
Alex Ling
adf260bc35 Bump version to v0.19.1 2021-01-27 06:33:45 +00:00
Alex Ling
432d6f0cd5 Run CI for hotfix/* branches 2021-01-27 06:33:45 +00:00
Alex Ling
3de314ae9a Fix incorrect background color on reader page 2021-01-27 06:33:45 +00:00
Alex Ling
c1c8cca877 Use Ameba to enforce max line width
Didn't know Ameba supports this!
2021-01-27 04:18:47 +00:00
Alex Ling
07965b98b7 Force File::Info#inode to return UInt64 2021-01-27 03:42:51 +00:00
Alex Ling
5779d225f6 Merge branch 'dev' of https://github.com/hkalexling/Mango into dev 2021-01-27 03:23:08 +00:00
Alex Ling
bf18a14016 Use inode number 2021-01-27 03:19:58 +00:00
Alex Ling
605dc61777 Merge pull request #150 from Leeingnyo/fix/allow-uppercase-extensions
Make file extension check case-insensitive
2021-01-26 19:16:12 +08:00
Alex Ling
def64d9f98 Rename interesting files to supported files 2021-01-26 10:55:50 +00:00
Leeingnyo
0ba2409c9a add tests about is_interesting_file 2021-01-26 04:18:09 +09:00
Leeingnyo
2b0cf41336 add and apply util method is_interesting_file 2021-01-26 04:17:32 +09:00
Leeingnyo
c51cb28df2 make filename extension downcase for comparing 2021-01-25 23:13:35 +09:00
Alex Ling
2b079c652d Fix duplicating options on the download page 2021-01-20 08:02:07 +00:00
Alex Ling
68050a9025 Fix incorrect dropdown color in dark mode 2021-01-20 05:20:03 +00:00
Alex Ling
54cd15d542 Mark items unavailable and retire DB optimization
This prepares us for the moving metadata to DB in the future
2021-01-19 15:09:38 +00:00
Alex Ling
781de97c68 Make thumbnail generation slower
This reduces the IO stress
2021-01-19 15:06:27 +00:00
Alex Ling
c7be0e0e7c Separate insert_id into titles and entries 2021-01-19 09:08:31 +00:00
Alex Ling
667d390be4 Signature matching 2021-01-19 08:43:45 +00:00
Alex Ling
7f76322377 Merge branch 'dev' into feature/signature 2021-01-18 06:54:38 +00:00
Alex Ling
377c4c6554 Stop the process when the server fails to start 2021-01-18 06:44:10 +00:00
Alex Ling
952aa0c6ca Fix linter 2021-01-17 15:59:42 +00:00
Alex Ling
bd81c2e005 Fix incorrect migration SQL 2021-01-17 15:58:13 +00:00
Alex Ling
b471ed2fa0 Upgrade MG 2021-01-17 15:49:10 +00:00
Alex Ling
7507ab64ad Bump version to v0.19.0 2021-01-17 08:34:35 +00:00
Alex Ling
e4587d36bc Fix linter 2021-01-17 08:25:01 +00:00
Alex Ling
7d6d3640ad Disable the tagging UI for non-admin users 2021-01-17 08:16:40 +00:00
Alex Ling
3071d44e32 Fix admin API bypassing 2021-01-17 08:10:43 +00:00
Alex Ling
7a09c9006a Set up foreign keys 2021-01-17 04:47:06 +00:00
Alex Ling
959560c7a7 Add titles and move insert_ids to class variable
This fixes the bug where the new ids are not saved
2021-01-17 04:45:55 +00:00
Alex Ling
ff679b30d8 Capitalize the UNIQUE keyword 2021-01-17 04:41:05 +00:00
Alex Ling
f7a360c2d8 Proper DB migration 2021-01-16 17:11:57 +00:00
Alex Ling
1065b430e3 Rewrite tagging UI with suggestions (#146) 2021-01-14 13:08:50 +00:00
Alex Ling
5abf7032a5 Use less 2021-01-14 13:04:57 +00:00
Alex Ling
18e8e88c66 Initial work on title signature 2021-01-14 08:23:39 +00:00
Alex Ling
44336c546a Bump version to v0.18.3 2021-01-12 10:14:12 +00:00
Alex Ling
a4c6e6611c Try WSS first, and fallback to WS (#144) 2021-01-12 10:13:06 +00:00
Alex Ling
0b457a2797 Merge branch 'master' of https://github.com/hkalexling/Mango 2021-01-11 15:37:34 +00:00
Alex Ling
653751bede Merge branch 'dev' 2021-01-11 15:37:06 +00:00
Alex Ling
a02bf4a81e Bump version to v0.18.2 2021-01-11 15:22:51 +00:00
Alex Ling
5271d12f4c Respect base URL in WS connections 2021-01-11 15:05:58 +00:00
Alex Ling
c2e2f0b9b3 Merge pull request #143 from hkalexling/all-contributors/add-h45h74x
docs: add h45h74x as a contributor
2021-01-11 19:32:47 +08:00
allcontributors[bot]
72d319902e docs: update .all-contributorsrc [skip ci] 2021-01-11 11:31:21 +00:00
allcontributors[bot]
bbd0fd68cb docs: update README.md [skip ci] 2021-01-11 11:31:20 +00:00
Alex Ling
0fb1e1598d Remove sourcerer.io HoF and use all-contributors
[skip ci]
RIP sourcerer.io https://github.com/sourcerer-io/sourcerer-app/issues/632
2021-01-11 11:28:30 +00:00
Alex Ling
4645582f5d Bump version to v0.18.1 2021-01-11 05:29:28 +00:00
Alex Ling
ac9c51dd33 Remove non-existing #root from css selectors (#142) 2021-01-11 05:28:44 +00:00
Alex Ling
f51d27860a Validate input index before flipping page 2021-01-09 15:49:34 +00:00
53 changed files with 1460 additions and 357 deletions

102
.all-contributorsrc Normal file
View File

@@ -0,0 +1,102 @@
{
"projectName": "Mango",
"projectOwner": "hkalexling",
"repoType": "github",
"repoHost": "https://github.com",
"files": [
"README.md"
],
"imageSize": 100,
"commit": false,
"commitConvention": "none",
"contributors": [
{
"login": "hkalexling",
"name": "Alex Ling",
"avatar_url": "https://avatars1.githubusercontent.com/u/7845831?v=4",
"profile": "https://github.com/hkalexling/",
"contributions": [
"code",
"doc",
"infra"
]
},
{
"login": "jaredlt",
"name": "jaredlt",
"avatar_url": "https://avatars1.githubusercontent.com/u/8590311?v=4",
"profile": "https://github.com/jaredlt",
"contributions": [
"code",
"ideas",
"design"
]
},
{
"login": "shincurry",
"name": "ココロ",
"avatar_url": "https://avatars1.githubusercontent.com/u/4946624?v=4",
"profile": "https://windisco.com/",
"contributions": [
"infra"
]
},
{
"login": "noirscape",
"name": "Valentijn",
"avatar_url": "https://avatars0.githubusercontent.com/u/13433513?v=4",
"profile": "https://catgirlsin.space/",
"contributions": [
"infra"
]
},
{
"login": "flying-sausages",
"name": "flying-sausages",
"avatar_url": "https://avatars1.githubusercontent.com/u/23618693?v=4",
"profile": "https://github.com/flying-sausages",
"contributions": [
"doc",
"ideas"
]
},
{
"login": "XavierSchiller",
"name": "Xavier",
"avatar_url": "https://avatars1.githubusercontent.com/u/22575255?v=4",
"profile": "https://github.com/XavierSchiller",
"contributions": [
"infra"
]
},
{
"login": "WROIATE",
"name": "Jarao",
"avatar_url": "https://avatars3.githubusercontent.com/u/44677306?v=4",
"profile": "https://github.com/WROIATE",
"contributions": [
"infra"
]
},
{
"login": "Leeingnyo",
"name": "이인용",
"avatar_url": "https://avatars0.githubusercontent.com/u/6760150?v=4",
"profile": "https://github.com/Leeingnyo",
"contributions": [
"code"
]
},
{
"login": "h45h74x",
"name": "Simon",
"avatar_url": "https://avatars1.githubusercontent.com/u/27204033?v=4",
"profile": "http://h45h74x.eu.org",
"contributions": [
"code"
]
}
],
"contributorsPerLine": 7,
"skipCi": true
}

View File

@@ -7,3 +7,8 @@ Lint/UnusedArgument:
- src/routes/*
Metrics/CyclomaticComplexity:
Enabled: false
Layout/LineLength:
Enabled: true
MaxLength: 80
Excluded:
- src/routes/api.cr

6
.github/autoapproval.yml vendored Normal file
View File

@@ -0,0 +1,6 @@
from_owner:
- hkalexling
required_labels:
- autoapprove
apply_labels:
- autoapproved

View File

@@ -2,7 +2,7 @@ name: Build
on:
push:
branches: [ master, dev ]
branches: [ master, dev, hotfix/* ]
pull_request:
branches: [ master, dev ]

1
.gitignore vendored
View File

@@ -12,3 +12,4 @@ mango
public/css/uikit.css
public/img/*.svg
public/js/*.min.js
public/css/*.css

View File

@@ -29,7 +29,6 @@ test:
check:
crystal tool format --check
./bin/ameba
./dev/linewidth.sh
arm32v7:
crystal build src/mango.cr --release --progress --error-trace --cross-compile --target='arm-linux-gnueabihf' -o mango-arm32v7

View File

@@ -52,7 +52,7 @@ The official docker images are available on [Dockerhub](https://hub.docker.com/r
### CLI
```
Mango - Manga Server and Web Reader. Version 0.18.0
Mango - Manga Server and Web Reader. Version 0.20.0
Usage:
@@ -82,7 +82,6 @@ library_path: ~/mango/library
db_path: ~/mango/mango.db
scan_interval_minutes: 5
thumbnail_generation_interval_hours: 24
db_optimization_interval_hours: 24
log_level: info
upload_path: ~/mango/uploads
plugin_path: ~/mango/plugins
@@ -157,5 +156,26 @@ Mobile UI:
## Contributors
Please check the [development guideline](https://github.com/hkalexling/Mango/wiki/Development) if you are interested in code contributions.
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<table>
<tr>
<td align="center"><a href="https://github.com/hkalexling/"><img src="https://avatars1.githubusercontent.com/u/7845831?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Alex Ling</b></sub></a><br /><a href="https://github.com/hkalexling/Mango/commits?author=hkalexling" title="Code">💻</a> <a href="https://github.com/hkalexling/Mango/commits?author=hkalexling" title="Documentation">📖</a> <a href="#infra-hkalexling" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
<td align="center"><a href="https://github.com/jaredlt"><img src="https://avatars1.githubusercontent.com/u/8590311?v=4?s=100" width="100px;" alt=""/><br /><sub><b>jaredlt</b></sub></a><br /><a href="https://github.com/hkalexling/Mango/commits?author=jaredlt" title="Code">💻</a> <a href="#ideas-jaredlt" title="Ideas, Planning, & Feedback">🤔</a> <a href="#design-jaredlt" title="Design">🎨</a></td>
<td align="center"><a href="https://windisco.com/"><img src="https://avatars1.githubusercontent.com/u/4946624?v=4?s=100" width="100px;" alt=""/><br /><sub><b>ココロ</b></sub></a><br /><a href="#infra-shincurry" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
<td align="center"><a href="https://catgirlsin.space/"><img src="https://avatars0.githubusercontent.com/u/13433513?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Valentijn</b></sub></a><br /><a href="#infra-noirscape" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
<td align="center"><a href="https://github.com/flying-sausages"><img src="https://avatars1.githubusercontent.com/u/23618693?v=4?s=100" width="100px;" alt=""/><br /><sub><b>flying-sausages</b></sub></a><br /><a href="https://github.com/hkalexling/Mango/commits?author=flying-sausages" title="Documentation">📖</a> <a href="#ideas-flying-sausages" title="Ideas, Planning, & Feedback">🤔</a></td>
<td align="center"><a href="https://github.com/XavierSchiller"><img src="https://avatars1.githubusercontent.com/u/22575255?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Xavier</b></sub></a><br /><a href="#infra-XavierSchiller" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
<td align="center"><a href="https://github.com/WROIATE"><img src="https://avatars3.githubusercontent.com/u/44677306?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Jarao</b></sub></a><br /><a href="#infra-WROIATE" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
</tr>
<tr>
<td align="center"><a href="https://github.com/Leeingnyo"><img src="https://avatars0.githubusercontent.com/u/6760150?v=4?s=100" width="100px;" alt=""/><br /><sub><b>이인용</b></sub></a><br /><a href="https://github.com/hkalexling/Mango/commits?author=Leeingnyo" title="Code">💻</a></td>
<td align="center"><a href="http://h45h74x.eu.org"><img src="https://avatars1.githubusercontent.com/u/27204033?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Simon</b></sub></a><br /><a href="https://github.com/hkalexling/Mango/commits?author=h45h74x" title="Code">💻</a></td>
</tr>
</table>
[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/0)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/0)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/1)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/1)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/2)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/2)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/3)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/3)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/4)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/4)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/5)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/5)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/6)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/6)[![](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/images/7)](https://sourcerer.io/fame/hkalexling/hkalexling/Mango/links/7)
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->

View File

@@ -1,5 +0,0 @@
#!/bin/sh
[ ! -z "$(grep '.\{80\}' --exclude-dir=lib --include="*.cr" -nr --color=always . | grep -v "routes/api.cr" | tee /dev/tty)" ] \
&& echo "The above lines exceed the 80 characters limit" \
|| exit 0

View File

@@ -0,0 +1,85 @@
class ForeignKeys < MG::Base
def up : String
<<-SQL
-- add foreign key to tags
ALTER TABLE tags RENAME TO tmp;
CREATE TABLE tags (
id TEXT NOT NULL,
tag TEXT NOT NULL,
UNIQUE (id, tag),
FOREIGN KEY (id) REFERENCES titles (id)
ON UPDATE CASCADE
ON DELETE CASCADE
);
INSERT INTO tags
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE INDEX tags_id_idx ON tags (id);
CREATE INDEX tags_tag_idx ON tags (tag);
-- add foreign key to thumbnails
ALTER TABLE thumbnails RENAME TO tmp;
CREATE TABLE thumbnails (
id TEXT NOT NULL,
data BLOB NOT NULL,
filename TEXT NOT NULL,
mime TEXT NOT NULL,
size INTEGER NOT NULL,
FOREIGN KEY (id) REFERENCES ids (id)
ON UPDATE CASCADE
ON DELETE CASCADE
);
INSERT INTO thumbnails
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE UNIQUE INDEX tn_index ON thumbnails (id);
SQL
end
def down : String
<<-SQL
-- remove foreign key from thumbnails
ALTER TABLE thumbnails RENAME TO tmp;
CREATE TABLE thumbnails (
id TEXT NOT NULL,
data BLOB NOT NULL,
filename TEXT NOT NULL,
mime TEXT NOT NULL,
size INTEGER NOT NULL
);
INSERT INTO thumbnails
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE UNIQUE INDEX tn_index ON thumbnails (id);
-- remove foreign key from tags
ALTER TABLE tags RENAME TO tmp;
CREATE TABLE tags (
id TEXT NOT NULL,
tag TEXT NOT NULL,
UNIQUE (id, tag)
);
INSERT INTO tags
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE INDEX tags_id_idx ON tags (id);
CREATE INDEX tags_tag_idx ON tags (tag);
SQL
end
end

19
migration/ids.2.cr Normal file
View File

@@ -0,0 +1,19 @@
class CreateIds < MG::Base
def up : String
<<-SQL
CREATE TABLE IF NOT EXISTS ids (
path TEXT NOT NULL,
id TEXT NOT NULL,
is_title INTEGER NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS path_idx ON ids (path);
CREATE UNIQUE INDEX IF NOT EXISTS id_idx ON ids (id);
SQL
end
def down : String
<<-SQL
DROP TABLE ids;
SQL
end
end

View File

@@ -0,0 +1,50 @@
class IDSignature < MG::Base
def up : String
<<-SQL
ALTER TABLE ids ADD COLUMN signature TEXT;
SQL
end
def down : String
<<-SQL
-- remove signature column from ids
ALTER TABLE ids RENAME TO tmp;
CREATE TABLE ids (
path TEXT NOT NULL,
id TEXT NOT NULL
);
INSERT INTO ids
SELECT path, id
FROM tmp;
DROP TABLE tmp;
-- recreate the indices
CREATE UNIQUE INDEX path_idx ON ids (path);
CREATE UNIQUE INDEX id_idx ON ids (id);
-- recreate the foreign key constraint on thumbnails
ALTER TABLE thumbnails RENAME TO tmp;
CREATE TABLE thumbnails (
id TEXT NOT NULL,
data BLOB NOT NULL,
filename TEXT NOT NULL,
mime TEXT NOT NULL,
size INTEGER NOT NULL,
FOREIGN KEY (id) REFERENCES ids (id)
ON UPDATE CASCADE
ON DELETE CASCADE
);
INSERT INTO thumbnails
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE UNIQUE INDEX tn_index ON thumbnails (id);
SQL
end
end

View File

@@ -0,0 +1,33 @@
class RelativePath < MG::Base
def up : String
base = Config.current.library_path
# Escape single quotes in case the path contains them, and remove the
# trailing slash (this is a mistake, fixed in DB version 10)
base = base.gsub("'", "''").rstrip "/"
<<-SQL
-- update the path column in ids to relative paths
UPDATE ids
SET path = REPLACE(path, '#{base}', '');
-- update the path column in titles to relative paths
UPDATE titles
SET path = REPLACE(path, '#{base}', '');
SQL
end
def down : String
base = Config.current.library_path
base = base.gsub("'", "''").rstrip "/"
<<-SQL
-- update the path column in ids to absolute paths
UPDATE ids
SET path = '#{base}' || path;
-- update the path column in titles to absolute paths
UPDATE titles
SET path = '#{base}' || path;
SQL
end
end

View File

@@ -0,0 +1,31 @@
# In DB version 8, we replaced the absolute paths in DB with relative paths,
# but we mistakenly left the starting slashes. This migration removes them.
class RelativePathFix < MG::Base
def up : String
<<-SQL
-- remove leading slashes from the paths in ids
UPDATE ids
SET path = SUBSTR(path, 2, LENGTH(path) - 1)
WHERE path LIKE '/%';
-- remove leading slashes from the paths in titles
UPDATE titles
SET path = SUBSTR(path, 2, LENGTH(path) - 1)
WHERE path LIKE '/%';
SQL
end
def down : String
<<-SQL
-- add leading slashes to paths in ids
UPDATE ids
SET path = '/' || path
WHERE path NOT LIKE '/%';
-- add leading slashes to paths in titles
UPDATE titles
SET path = '/' || path
WHERE path NOT LIKE '/%';
SQL
end
end

19
migration/tags.4.cr Normal file
View File

@@ -0,0 +1,19 @@
class CreateTags < MG::Base
def up : String
<<-SQL
CREATE TABLE IF NOT EXISTS tags (
id TEXT NOT NULL,
tag TEXT NOT NULL,
UNIQUE (id, tag)
);
CREATE INDEX IF NOT EXISTS tags_id_idx ON tags (id);
CREATE INDEX IF NOT EXISTS tags_tag_idx ON tags (tag);
SQL
end
def down : String
<<-SQL
DROP TABLE tags;
SQL
end
end

20
migration/thumbnails.3.cr Normal file
View File

@@ -0,0 +1,20 @@
class CreateThumbnails < MG::Base
def up : String
<<-SQL
CREATE TABLE IF NOT EXISTS thumbnails (
id TEXT NOT NULL,
data BLOB NOT NULL,
filename TEXT NOT NULL,
mime TEXT NOT NULL,
size INTEGER NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS tn_index ON thumbnails (id);
SQL
end
def down : String
<<-SQL
DROP TABLE thumbnails;
SQL
end
end

56
migration/titles.5.cr Normal file
View File

@@ -0,0 +1,56 @@
class CreateTitles < MG::Base
def up : String
<<-SQL
-- create titles
CREATE TABLE titles (
id TEXT NOT NULL,
path TEXT NOT NULL,
signature TEXT
);
CREATE UNIQUE INDEX titles_id_idx on titles (id);
CREATE UNIQUE INDEX titles_path_idx on titles (path);
-- migrate data from ids to titles
INSERT INTO titles
SELECT id, path, null
FROM ids
WHERE is_title = 1;
DELETE FROM ids
WHERE is_title = 1;
-- remove the is_title column from ids
ALTER TABLE ids RENAME TO tmp;
CREATE TABLE ids (
path TEXT NOT NULL,
id TEXT NOT NULL
);
INSERT INTO ids
SELECT path, id
FROM tmp;
DROP TABLE tmp;
-- recreate the indices
CREATE UNIQUE INDEX path_idx ON ids (path);
CREATE UNIQUE INDEX id_idx ON ids (id);
SQL
end
def down : String
<<-SQL
-- insert the is_title column
ALTER TABLE ids ADD COLUMN is_title INTEGER NOT NULL DEFAULT 0;
-- migrate data from titles to ids
INSERT INTO ids
SELECT path, id, 1
FROM titles;
-- remove titles
DROP TABLE titles;
SQL
end
end

View File

@@ -0,0 +1,94 @@
class UnavailableIDs < MG::Base
def up : String
<<-SQL
-- add unavailable column to ids
ALTER TABLE ids ADD COLUMN unavailable INTEGER NOT NULL DEFAULT 0;
-- add unavailable column to titles
ALTER TABLE titles ADD COLUMN unavailable INTEGER NOT NULL DEFAULT 0;
SQL
end
def down : String
<<-SQL
-- remove unavailable column from ids
ALTER TABLE ids RENAME TO tmp;
CREATE TABLE ids (
path TEXT NOT NULL,
id TEXT NOT NULL,
signature TEXT
);
INSERT INTO ids
SELECT path, id, signature
FROM tmp;
DROP TABLE tmp;
-- recreate the indices
CREATE UNIQUE INDEX path_idx ON ids (path);
CREATE UNIQUE INDEX id_idx ON ids (id);
-- recreate the foreign key constraint on thumbnails
ALTER TABLE thumbnails RENAME TO tmp;
CREATE TABLE thumbnails (
id TEXT NOT NULL,
data BLOB NOT NULL,
filename TEXT NOT NULL,
mime TEXT NOT NULL,
size INTEGER NOT NULL,
FOREIGN KEY (id) REFERENCES ids (id)
ON UPDATE CASCADE
ON DELETE CASCADE
);
INSERT INTO thumbnails
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE UNIQUE INDEX tn_index ON thumbnails (id);
-- remove unavailable column from titles
ALTER TABLE titles RENAME TO tmp;
CREATE TABLE titles (
id TEXT NOT NULL,
path TEXT NOT NULL,
signature TEXT
);
INSERT INTO titles
SELECT path, id, signature
FROM tmp;
DROP TABLE tmp;
-- recreate the indices
CREATE UNIQUE INDEX titles_id_idx on titles (id);
CREATE UNIQUE INDEX titles_path_idx on titles (path);
-- recreate the foreign key constraint on tags
ALTER TABLE tags RENAME TO tmp;
CREATE TABLE tags (
id TEXT NOT NULL,
tag TEXT NOT NULL,
UNIQUE (id, tag),
FOREIGN KEY (id) REFERENCES titles (id)
ON UPDATE CASCADE
ON DELETE CASCADE
);
INSERT INTO tags
SELECT * FROM tmp;
DROP TABLE tmp;
CREATE INDEX tags_id_idx ON tags (id);
CREATE INDEX tags_tag_idx ON tags (tag);
SQL
end
end

20
migration/users.1.cr Normal file
View File

@@ -0,0 +1,20 @@
class CreateUsers < MG::Base
def up : String
<<-SQL
CREATE TABLE IF NOT EXISTS users (
username TEXT NOT NULL,
password TEXT NOT NULL,
token TEXT,
admin INTEGER NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS username_idx ON users (username);
CREATE UNIQUE INDEX IF NOT EXISTS token_idx ON users (token);
SQL
end
def down : String
<<-SQL
DROP TABLE users;
SQL
end
end

View File

@@ -7,6 +7,7 @@
"license": "MIT",
"devDependencies": {
"@babel/preset-env": "^7.11.5",
"all-contributors-cli": "^6.19.0",
"gulp": "^4.0.2",
"gulp-babel": "^8.0.0",
"gulp-babel-minify": "^0.5.1",

View File

@@ -1,146 +0,0 @@
.uk-alert-close {
color: black !important;
}
.uk-card-body {
padding: 20px;
}
.uk-card-media-top {
width: 100%;
height: 250px;
}
@media (min-width: 600px) {
.uk-card-media-top {
height: 300px;
}
}
.uk-card-media-top>img {
height: 100%;
width: 100%;
object-fit: cover;
}
.uk-card-title {
max-height: 3em;
}
.acard:hover {
cursor: pointer;
}
.reader-bg {
background-color: black;
}
.break-word {
word-wrap: break-word;
}
.uk-logo>img {
height: 90px;
width: 90px;
}
.uk-search {
width: 100%;
}
#selectable .ui-selecting {
background: #EEE6B9;
}
#selectable .ui-selected {
background: #F4E487;
}
.uk-light #selectable .ui-selecting {
background: #5E5731;
}
.uk-light #selectable .ui-selected {
background: #9D9252;
}
td>.uk-dropdown {
white-space: pre-line;
}
#edit-modal .uk-grid>div {
height: 300px;
}
#edit-modal #cover {
height: 100%;
width: 100%;
object-fit: cover;
}
#edit-modal #cover-upload {
height: 100%;
box-sizing: border-box;
}
#edit-modal .uk-modal-body .uk-inline {
width: 100%;
}
.item .uk-card-title {
font-size: 1rem;
}
.grayscale {
filter: grayscale(100%);
}
.uk-light .uk-navbar-dropdown,
.uk-light .uk-modal-header,
.uk-light .uk-modal-body,
.uk-light .uk-modal-footer {
background: #222;
}
.uk-light .uk-dropdown {
background: #333;
}
.uk-light .uk-navbar-dropdown,
.uk-light .uk-dropdown {
color: #ccc;
}
.uk-light .uk-nav-header,
.uk-light .uk-description-list>dt {
color: #555;
}
[x-cloak] {
display: none;
}
#select-bar-controls a {
transform: scale(1.5, 1.5);
}
#select-bar-controls a:hover {
color: orange;
}
#main-section {
position: relative;
}
#totop-wrapper {
position: absolute;
top: 100vh;
right: 2em;
bottom: 0;
}
#totop-wrapper a {
position: fixed;
position: sticky;
top: calc(100vh - 5em);
}

124
public/css/mango.less Normal file
View File

@@ -0,0 +1,124 @@
// Item cards
.item .uk-card {
cursor: pointer;
.uk-card-media-top {
width: 100%;
height: 250px;
@media (min-width: 600px) {
height: 300px;
}
img {
height: 100%;
width: 100%;
object-fit: cover;
&.grayscale {
filter: grayscale(100%);
}
}
}
.uk-card-body {
padding: 20px;
.uk-card-title {
max-height: 3em;
font-size: 1rem;
}
}
}
// jQuery selectable
#selectable {
.ui-selecting {
background: #EEE6B9;
}
.ui-selected {
background: #F4E487;
}
.uk-light & {
.ui-selecting {
background: #5E5731;
}
.ui-selected {
background: #9D9252;
}
}
}
// Edit modal
#edit-modal {
.uk-grid > div {
height: 300px;
}
#cover {
height: 100%;
width: 100%;
object-fit: cover;
}
#cover-upload {
height: 100%;
box-sizing: border-box;
}
.uk-modal-body .uk-inline {
width: 100%;
}
}
// Dark theme
.uk-light {
.uk-modal-header,
.uk-modal-body,
.uk-modal-footer {
background: #222;
}
.uk-navbar-dropdown,
.uk-dropdown {
color: #ccc;
background: #333;
}
.uk-nav-header,
.uk-description-list > dt {
color: #555;
}
}
// Alpine magic
[x-cloak] {
display: none;
}
// Batch select bar on title page
#select-bar-controls {
a {
transform: scale(1.5, 1.5);
&:hover {
color: orange;
}
}
}
// Totop button
#totop-wrapper {
position: absolute;
top: 100vh;
right: 2em;
bottom: 0;
a {
position: fixed;
position: sticky;
top: calc(100vh - 5em);
}
}
// Misc
.uk-alert-close {
color: black !important;
}
.break-word {
word-wrap: break-word;
}
.uk-search {
width: 100%;
}

58
public/css/tags.less Normal file
View File

@@ -0,0 +1,58 @@
@light-gray: #e5e5e5;
@gray: #666666;
@black: #141414;
@blue: rgb(30, 135, 240);
@white1: rgba(255, 255, 255, .1);
@white2: rgba(255, 255, 255, .2);
@white7: rgba(255, 255, 255, .7);
.select2-container--default {
.select2-selection--multiple {
border: 1px solid @light-gray;
.select2-selection__choice,
.select2-selection__choice__remove,
.select2-selection__choice__remove:hover
{
background-color: @blue;
color: white;
border: none;
border-radius: 2px;
}
}
.select2-dropdown {
.select2-results__option--highlighted.select2-results__option--selectable {
background-color: @blue;
}
.select2-results__option--selected:not(.select2-results__option--highlighted) {
background-color: @light-gray
}
}
}
.uk-light {
.select2-container--default {
.select2-selection {
background-color: @white1;
}
.select2-selection--multiple {
border: 1px solid @white2;
.select2-selection__choice,
.select2-selection__choice__remove,
.select2-selection__choice__remove:hover
{
background-color: white;
color: @gray;
border: none;
}
.select2-search__field {
color: @white7;
}
}
}
.select2-dropdown {
background-color: @black;
.select2-results__option--selected:not(.select2-results__option--highlighted) {
background-color: @white2;
}
}
}

View File

@@ -4,21 +4,30 @@ const component = () => {
paused: undefined,
loading: false,
toggling: false,
ws: undefined,
init() {
const ws = new WebSocket(`ws://${location.host}/api/admin/mangadex/queue`);
ws.onmessage = event => {
wsConnect(secure = true) {
const url = `${secure ? 'wss' : 'ws'}://${location.host}${base_url}api/admin/mangadex/queue`;
console.log(`Connecting to ${url}`);
this.ws = new WebSocket(url);
this.ws.onmessage = event => {
const data = JSON.parse(event.data);
this.jobs = data.jobs;
this.paused = data.paused;
};
ws.onerror = err => {
alert('danger', `Socket connection failed. Error: ${err}`);
this.ws.onclose = () => {
if (this.ws.failed)
return this.wsConnect(false);
alert('danger', 'Socket connection closed');
};
ws.onclose = err => {
this.ws.onerror = () => {
if (secure)
return this.ws.failed = true;
alert('danger', 'Socket connection failed');
};
},
init() {
this.wsConnect();
this.load();
},
load() {

View File

@@ -156,8 +156,8 @@ const search = () => {
langs.unshift('All');
group_names.unshift('All');
$('select#lang-select').append(langs.map(e => `<option>${e}</option>`).join(''));
$('select#group-select').append(group_names.map(e => `<option>${e}</option>`).join(''));
$('select#lang-select').html(langs.map(e => `<option>${e}</option>`).join(''));
$('select#group-select').html(group_names.map(e => `<option>${e}</option>`).join(''));
$('#filter-form').removeAttr('hidden');
@@ -241,7 +241,7 @@ const buildTable = () => {
if (v === 'All') return;
if (k === 'group') {
chapters = chapters.filter(c => {
unescaped_groups = Object.entries(c.groups).map(([g, id]) => unescapeHTML(g));
const unescaped_groups = Object.entries(c.groups).map(([g, id]) => unescapeHTML(g));
return unescaped_groups.indexOf(v) >= 0;
});
return;

View File

@@ -0,0 +1,60 @@
const component = () => {
return {
empty: true,
titles: [],
entries: [],
loading: true,
load() {
this.loading = true;
this.request('GET', `${base_url}api/admin/titles/missing`, data => {
this.titles = data.titles;
this.request('GET', `${base_url}api/admin/entries/missing`, data => {
this.entries = data.entries;
this.loading = false;
this.empty = this.entries.length === 0 && this.titles.length === 0;
});
});
},
rm(event) {
const rawID = event.currentTarget.closest('tr').id;
const [type, id] = rawID.split('-');
const url = `${base_url}api/admin/${type === 'title' ? 'titles' : 'entries'}/missing/${id}`;
this.request('DELETE', url, () => {
this.load();
});
},
rmAll() {
UIkit.modal.confirm('Are you sure? All metadata associated with these items, including their tags and thumbnails, will be deleted from the database.', {
labels: {
ok: 'Yes, delete them',
cancel: 'Cancel'
}
}).then(() => {
this.request('DELETE', `${base_url}api/admin/titles/missing`, () => {
this.request('DELETE', `${base_url}api/admin/entries/missing`, () => {
this.load();
});
});
});
},
request(method, url, cb) {
console.log(url);
$.ajax({
type: method,
url: url,
contentType: 'application/json'
})
.done(data => {
if (data.error) {
alert('danger', `Failed to ${method} ${url}. Error: ${data.error}`);
return;
}
if (cb) cb(data);
})
.fail((jqXHR, status) => {
alert('danger', `Failed to ${method} ${url}. Error: [${jqXHR.status}] ${jqXHR.statusText}`);
});
}
};
};

View File

@@ -101,8 +101,9 @@ const readerComponent = () => {
*/
flipPage(isNext) {
const idx = parseInt(this.curItem.id);
const delta = isNext ? 1 : -1;
const newIdx = idx + delta;
const newIdx = idx + (isNext ? 1 : -1);
if (newIdx <= 0 || newIdx > this.items.length) return;
this.toPage(newIdx);
@@ -240,7 +241,7 @@ const readerComponent = () => {
*/
setupScroller() {
if (this.mode !== 'continuous') return;
$('#root img').each((idx, el) => {
$('img').each((idx, el) => {
$(el).on('inview', (event, inView) => {
if (inView) {
const current = $(event.currentTarget).attr('id');

View File

@@ -255,48 +255,65 @@ const bulkProgress = (action, el) => {
const tagsComponent = () => {
return {
loading: true,
isAdmin: false,
tags: [],
newTag: '',
inputShown: false,
tid: $('.upload-field').attr('data-title-id'),
loading: true,
load(admin) {
this.isAdmin = admin;
const url = `${base_url}api/tags/${this.tid}`;
this.request(url, 'GET', (data) => {
this.tags = data.tags;
this.loading = false;
$('.tag-select').select2({
tags: true,
placeholder: this.isAdmin ? 'Tag the title' : 'No tags found',
disabled: !this.isAdmin,
templateSelection(state) {
const a = document.createElement('a');
a.setAttribute('href', `${base_url}tags/${encodeURIComponent(state.text)}`);
a.setAttribute('class', 'uk-link-reset');
a.onclick = event => {
event.stopPropagation();
};
a.innerText = state.text;
return a;
}
});
},
add() {
const tag = this.newTag.trim();
const url = `${base_url}api/admin/tags/${this.tid}/${encodeURIComponent(tag)}`;
this.request(url, 'PUT', () => {
this.tags.push(tag);
this.newTag = '';
});
},
keydown(event) {
if (event.key === 'Enter')
this.add()
},
rm(event) {
const tag = event.currentTarget.id.split('-')[0];
const url = `${base_url}api/admin/tags/${this.tid}/${encodeURIComponent(tag)}`;
this.request(url, 'DELETE', () => {
const idx = this.tags.indexOf(tag);
if (idx < 0) return;
this.tags.splice(idx, 1);
});
},
toggleInput(nextTick) {
this.inputShown = !this.inputShown;
if (this.inputShown) {
nextTick(() => {
$('#tag-input').get(0).focus();
this.request(`${base_url}api/tags`, 'GET', (data) => {
const allTags = data.tags;
const url = `${base_url}api/tags/${this.tid}`;
this.request(url, 'GET', data => {
this.tags = data.tags;
allTags.forEach(t => {
const op = new Option(t, t, false, this.tags.indexOf(t) >= 0);
$('.tag-select').append(op);
});
$('.tag-select').on('select2:select', e => {
this.onAdd(e);
});
$('.tag-select').on('select2:unselect', e => {
this.onDelete(e);
});
$('.tag-select').on('change', () => {
this.onChange();
});
$('.tag-select').trigger('change');
this.loading = false;
});
}
});
},
onChange() {
this.tags = $('.tag-select').select2('data').map(o => o.text);
},
onAdd(event) {
const tag = event.params.data.text;
const url = `${base_url}api/admin/tags/${this.tid}/${encodeURIComponent(tag)}`;
this.request(url, 'PUT');
},
onDelete(event) {
const tag = event.params.data.text;
const url = `${base_url}api/admin/tags/${this.tid}/${encodeURIComponent(tag)}`;
this.request(url, 'DELETE');
},
request(url, method, cb) {
$.ajax({
@@ -305,9 +322,9 @@ const tagsComponent = () => {
dataType: 'json'
})
.done(data => {
if (data.success)
cb(data);
else {
if (data.success) {
if (cb) cb(data);
} else {
alert('danger', data.error);
}
})

View File

@@ -52,6 +52,10 @@ shards:
git: https://github.com/hkalexling/koa.git
version: 0.5.0
mg:
git: https://github.com/hkalexling/mg.git
version: 0.2.0+git.commit.171c46489d991a8353818e00fc6a3c4e0809ded9
myhtml:
git: https://github.com/kostya/myhtml.git
version: 1.5.1

View File

@@ -1,5 +1,5 @@
name: mango
version: 0.18.0
version: 0.20.0
authors:
- Alex Ling <hkalexling@gmail.com>
@@ -41,3 +41,5 @@ dependencies:
github: hkalexling/koa
tallboy:
github: epoch/tallboy
mg:
github: hkalexling/mg

View File

@@ -35,6 +35,23 @@ describe "compare_numerically" do
end
end
describe "is_supported_file" do
it "returns true when the filename has a supported extension" do
filename = "manga.cbz"
is_supported_file(filename).should eq true
end
it "returns true when the filename does not have a supported extension" do
filename = "info.json"
is_supported_file(filename).should eq false
end
it "is case insensitive" do
filename = "manga.ZiP"
is_supported_file(filename).should eq true
end
end
describe "chapter_sort" do
it "sorts correctly" do
ary = ["Vol.1 Ch.01", "Vol.1 Ch.02", "Vol.2 Ch. 2.5", "Ch. 3", "Ch.04"]

View File

@@ -13,7 +13,6 @@ class Config
property db_path : String = File.expand_path "~/mango/mango.db", home: true
property scan_interval_minutes : Int32 = 5
property thumbnail_generation_interval_hours : Int32 = 24
property db_optimization_interval_hours : Int32 = 24
property log_level : String = "info"
property upload_path : String = File.expand_path "~/mango/uploads",
home: true

View File

@@ -82,7 +82,12 @@ class AuthHandler < Kemal::Handler
if env.session.string? "token"
should_reject = !validate_token_admin(env)
end
env.response.status_code = 403 if should_reject
if should_reject
env.response.status_code = 403
send_error_page "HTTP 403: You are not authorized to visit " \
"#{env.request.path}"
return
end
end
call_next env

View File

@@ -11,13 +11,13 @@ class Entry
@title = File.basename @zip_path, File.extname @zip_path
@encoded_title = URI.encode @title
@size = (File.size @zip_path).humanize_bytes
id = storage.get_id @zip_path, false
id = storage.get_entry_id @zip_path, File.signature(@zip_path)
if id.nil?
id = random_str
storage.insert_id({
path: @zip_path,
id: id,
is_title: false,
storage.insert_entry_id({
path: @zip_path,
id: id,
signature: File.signature(@zip_path).to_s,
})
end
@id = id

View File

@@ -42,16 +42,6 @@ class Library
end
end
end
db_interval = Config.current.db_optimization_interval_hours
unless db_interval < 1
spawn do
loop do
Storage.default.optimize
sleep db_interval.hours
end
end
end
end
def titles
@@ -119,6 +109,7 @@ class Library
storage.close
Logger.debug "Scan completed"
Storage.default.mark_unavailable
end
def get_continue_reading_entries(username)
@@ -241,7 +232,7 @@ class Library
e.generate_thumbnail
# Sleep after each generation to minimize the impact on disk IO
# and CPU
sleep 0.5.seconds
sleep 1.seconds
end
@thumbnails_count += 1
end

View File

@@ -3,19 +3,20 @@ require "../archive"
class Title
getter dir : String, parent_id : String, title_ids : Array(String),
entries : Array(Entry), title : String, id : String,
encoded_title : String, mtime : Time
encoded_title : String, mtime : Time, signature : UInt64
@entry_display_name_cache : Hash(String, String)?
def initialize(@dir : String, @parent_id)
storage = Storage.default
id = storage.get_id @dir, true
@signature = Dir.signature dir
id = storage.get_title_id dir, signature
if id.nil?
id = random_str
storage.insert_id({
path: @dir,
id: id,
is_title: true,
storage.insert_title_id({
path: dir,
id: id,
signature: signature.to_s,
})
end
@id = id
@@ -35,7 +36,7 @@ class Title
@title_ids << title.id
next
end
if [".zip", ".cbz", ".rar", ".cbr"].includes? File.extname path
if is_supported_file path
entry = Entry.new path, self
@entries << entry if entry.pages > 0 || entry.err_msg
end
@@ -61,6 +62,7 @@ class Title
{% for str in ["dir", "title", "id"] %}
json.field {{str}}, @{{str.id}}
{% end %}
json.field "signature" { json.number @signature }
json.field "display_name", display_name
json.field "cover_url", cover_url
json.field "mtime" { json.number @mtime.to_unix }

View File

@@ -6,26 +6,14 @@ class Logger
SEVERITY_IDS = [0, 4, 5, 2, 3]
COLORS = [:light_cyan, :light_red, :red, :light_yellow, :light_magenta]
getter raw_log = Log.for ""
@@severity : Log::Severity = :info
use_default
def initialize
level = Config.current.log_level
{% begin %}
case level.downcase
when "off"
@@severity = :none
{% for lvl, i in LEVELS %}
when {{lvl}}
@@severity = Log::Severity.new SEVERITY_IDS[{{i}}]
{% end %}
else
raise "Unknown log level #{level}"
end
{% end %}
@log = Log.for("")
@@severity = Logger.get_severity
@backend = Log::IOBackend.new
format_proc = ->(entry : Log::Entry, io : IO) do
@@ -49,6 +37,24 @@ class Logger
Log.setup @@severity, @backend
end
def self.get_severity(level = "") : Log::Severity
if level.empty?
level = Config.current.log_level
end
{% begin %}
case level.downcase
when "off"
return Log::Severity::None
{% for lvl, i in LEVELS %}
when {{lvl}}
return Log::Severity.new SEVERITY_IDS[{{i}}]
{% end %}
else
raise "Unknown log level #{level}"
end
{% end %}
end
# Ignores @@severity and always log msg
def log(msg)
@backend.write Log::Entry.new "", Log::Severity::None, msg,
@@ -61,7 +67,7 @@ class Logger
{% for lvl in LEVELS %}
def {{lvl.id}}(msg)
@log.{{lvl.id}} { msg }
raw_log.{{lvl.id}} { msg }
end
def self.{{lvl.id}}(msg)
default.not_nil!.{{lvl.id}} msg

View File

@@ -8,7 +8,7 @@ require "option_parser"
require "clim"
require "tallboy"
MANGO_VERSION = "0.18.0"
MANGO_VERSION = "0.20.0"
# From http://www.network-science.de/ascii/
BANNER = %{
@@ -63,7 +63,12 @@ class CLI < Clim
Plugin::Downloader.default
spawn do
Server.new.start
begin
Server.new.start
rescue e
Logger.fatal e
Process.exit 1
end
end
MainFiber.start_and_block

View File

@@ -1,6 +1,9 @@
struct AdminRouter
def initialize
get "/admin" do |env|
storage = Storage.default
missing_count = storage.missing_titles.size +
storage.missing_entries.size
layout "admin"
end
@@ -66,5 +69,9 @@ struct AdminRouter
mangadex_base_url = Config.current.mangadex["base_url"]
layout "download-manager"
end
get "/admin/missing" do |env|
layout "missing-items"
end
end
end

View File

@@ -166,6 +166,21 @@ struct APIRouter
"error" => "string?",
}
Koa.object "missing", {
"path" => "string",
"id" => "string",
"signature" => "string",
}
Koa.array "missingAry", "$missing"
Koa.object "missingResult", {
"success" => "boolean",
"error" => "string?",
"entries" => "$missingAry?",
"titles" => "$missingAry?",
}
Koa.describe "Returns a page in a manga entry"
Koa.path "tid", desc: "Title ID"
Koa.path "eid", desc: "Entry ID"
@@ -713,6 +728,24 @@ struct APIRouter
end
end
Koa.describe "Returns all tags"
Koa.response 200, ref: "$tagsResult"
get "/api/tags" do |env|
begin
tags = Storage.default.list_tags
send_json env, {
"success" => true,
"tags" => tags,
}.to_json
rescue e
Logger.error e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
Koa.describe "Adds a new tag to a title"
Koa.path "tid", desc: "A title ID"
Koa.response 200, ref: "$result"
@@ -759,6 +792,120 @@ struct APIRouter
end
end
Koa.describe "Lists all missing titles"
Koa.response 200, ref: "$missingResult"
Koa.tag "admin"
get "/api/admin/titles/missing" do |env|
begin
send_json env, {
"success" => true,
"error" => nil,
"titles" => Storage.default.missing_titles,
}.to_json
rescue e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
Koa.describe "Lists all missing entries"
Koa.response 200, ref: "$missingResult"
Koa.tag "admin"
get "/api/admin/entries/missing" do |env|
begin
send_json env, {
"success" => true,
"error" => nil,
"entries" => Storage.default.missing_entries,
}.to_json
rescue e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
Koa.describe "Deletes all missing titles"
Koa.response 200, ref: "$result"
Koa.tag "admin"
delete "/api/admin/titles/missing" do |env|
begin
Storage.default.delete_missing_title
send_json env, {
"success" => true,
"error" => nil,
}.to_json
rescue e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
Koa.describe "Deletes all missing entries"
Koa.response 200, ref: "$result"
Koa.tag "admin"
delete "/api/admin/entries/missing" do |env|
begin
Storage.default.delete_missing_entry
send_json env, {
"success" => true,
"error" => nil,
}.to_json
rescue e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
Koa.describe "Deletes a missing title identified by `tid`", <<-MD
Does nothing if the given `tid` is not found or if the title is not missing.
MD
Koa.response 200, ref: "$result"
Koa.tag "admin"
delete "/api/admin/titles/missing/:tid" do |env|
begin
tid = env.params.url["tid"]
Storage.default.delete_missing_title tid
send_json env, {
"success" => true,
"error" => nil,
}.to_json
rescue e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
Koa.describe "Deletes a missing entry identified by `eid`", <<-MD
Does nothing if the given `eid` is not found or if the entry is not missing.
MD
Koa.response 200, ref: "$result"
Koa.tag "admin"
delete "/api/admin/entries/missing/:eid" do |env|
begin
eid = env.params.url["eid"]
Storage.default.delete_missing_entry eid
send_json env, {
"success" => true,
"error" => nil,
}.to_json
rescue e
send_json env, {
"success" => false,
"error" => e.message,
}.to_json
end
end
doc = Koa.generate
@@api_json = doc.to_json if doc

View File

@@ -154,6 +154,7 @@ struct MainRouter
end
get "/api" do |env|
base_url = Config.current.base_url
render "src/views/api.html.ecr"
end
end

View File

@@ -7,10 +7,6 @@ require "./routes/*"
class Server
def initialize
error 403 do |env|
message = "HTTP 403: You are not authorized to visit #{env.request.path}"
layout "message"
end
error 404 do |env|
message = "HTTP 404: Mango cannot find the page #{env.request.path}"
layout "message"

View File

@@ -3,6 +3,8 @@ require "crypto/bcrypt"
require "uuid"
require "base64"
require "./util/*"
require "mg"
require "../migration/*"
def hash_password(pw)
Crypto::Bcrypt::Password.create(pw).to_s
@@ -13,13 +15,16 @@ def verify_password(hash, pw)
end
class Storage
@@insert_entry_ids = [] of IDTuple
@@insert_title_ids = [] of IDTuple
@path : String
@db : DB::Database?
@insert_ids = [] of IDTuple
alias IDTuple = NamedTuple(path: String,
alias IDTuple = NamedTuple(
path: String,
id: String,
is_title: Bool)
signature: String?)
use_default
@@ -35,44 +40,15 @@ class Storage
MainFiber.run do
DB.open "sqlite3://#{@path}" do |db|
begin
# v0.18.0
db.exec "create table tags (id text, tag text, unique (id, tag))"
db.exec "create index tags_id_idx on tags (id)"
db.exec "create index tags_tag_idx on tags (tag)"
# v0.15.0
db.exec "create table thumbnails " \
"(id text, data blob, filename text, " \
"mime text, size integer)"
db.exec "create unique index tn_index on thumbnails (id)"
# v0.1.1
db.exec "create table ids" \
"(path text, id text, is_title integer)"
db.exec "create unique index path_idx on ids (path)"
db.exec "create unique index id_idx on ids (id)"
# v0.1.0
db.exec "create table users" \
"(username text, password text, token text, admin integer)"
MG::Migration.new(db, log: Logger.default.raw_log).migrate
rescue e
unless e.message.not_nil!.ends_with? "already exists"
Logger.fatal "Error when checking tables in DB: #{e}"
raise e
end
# If the DB is initialized through CLI but no user is added, we need
# to create the admin user when first starting the app
user_count = db.query_one "select count(*) from users", as: Int32
init_admin if init_user && user_count == 0
else
Logger.debug "Creating DB file at #{@path}"
db.exec "create unique index username_idx on users (username)"
db.exec "create unique index token_idx on users (token)"
init_admin if init_user
Logger.fatal "DB migration failed. #{e}"
raise e
end
user_count = db.query_one "select count(*) from users", as: Int32
init_admin if init_user && user_count == 0
# Verifies that the default username in config is valid
if Config.current.disable_login
username = Config.current.default_username
@@ -99,9 +75,11 @@ class Storage
private def get_db(&block : DB::Database ->)
if @db.nil?
DB.open "sqlite3://#{@path}" do |db|
db.exec "PRAGMA foreign_keys = 1"
yield db
end
else
@db.not_nil!.exec "PRAGMA foreign_keys = 1"
yield @db.not_nil!
end
end
@@ -254,32 +232,121 @@ class Storage
end
end
def get_id(path, is_title)
def get_title_id(path, signature)
id = nil
path = Path.new(path).relative_to(Config.current.library_path).to_s
MainFiber.run do
get_db do |db|
id = db.query_one? "select id from ids where path = (?)", path,
as: {String}
# First attempt to find the matching title in DB using BOTH path
# and signature
id = db.query_one? "select id from titles where path = (?) and " \
"signature = (?) and unavailable = 0",
path, signature.to_s, as: String
should_update = id.nil?
# If it fails, try to match using the path only. This could happen
# for example when a new entry is added to the title
id ||= db.query_one? "select id from titles where path = (?)", path,
as: String
# If it still fails, we will have to rely on the signature values.
# This could happen when the user moved or renamed the title, or
# a title containing the title
unless id
# If there are multiple rows with the same signature (this could
# happen simply by bad luck, or when the user copied a title),
# pick the row that has the most similar path to the give path
rows = [] of Tuple(String, String)
db.query "select id, path from titles where signature = (?)",
signature.to_s do |rs|
rs.each do
rows << {rs.read(String), rs.read(String)}
end
end
row = rows.max_by?(&.[1].components_similarity(path))
id = row[0] if row
end
# At this point, `id` would still be nil if there's no row matching
# either the path or the signature
# If we did identify a matching title, save the path and signature
# values back to the DB
if id && should_update
db.exec "update titles set path = (?), signature = (?), " \
"unavailable = 0 where id = (?)", path, signature.to_s, id
end
end
end
id
end
def insert_id(tp : IDTuple)
@insert_ids << tp
# See the comments in `#get_title_id` to see how this method works.
def get_entry_id(path, signature)
id = nil
path = Path.new(path).relative_to(Config.current.library_path).to_s
MainFiber.run do
get_db do |db|
id = db.query_one? "select id from ids where path = (?) and " \
"signature = (?) and unavailable = 0",
path, signature.to_s, as: String
should_update = id.nil?
id ||= db.query_one? "select id from ids where path = (?)", path,
as: String
unless id
rows = [] of Tuple(String, String)
db.query "select id, path from ids where signature = (?)",
signature.to_s do |rs|
rs.each do
rows << {rs.read(String), rs.read(String)}
end
end
row = rows.max_by?(&.[1].components_similarity(path))
id = row[0] if row
end
if id && should_update
db.exec "update ids set path = (?), signature = (?), " \
"unavailable = 0 where id = (?)", path, signature.to_s, id
end
end
end
id
end
def insert_entry_id(tp)
@@insert_entry_ids << tp
end
def insert_title_id(tp)
@@insert_title_ids << tp
end
def bulk_insert_ids
MainFiber.run do
get_db do |db|
db.transaction do |tx|
@insert_ids.each do |tp|
tx.connection.exec "insert into ids values (?, ?, ?)", tp[:path],
tp[:id], tp[:is_title] ? 1 : 0
db.transaction do |tran|
conn = tran.connection
@@insert_title_ids.each do |tp|
path = Path.new(tp[:path])
.relative_to(Config.current.library_path).to_s
conn.exec "insert into titles (id, path, signature, " \
"unavailable) values (?, ?, ?, 0)",
tp[:id], path, tp[:signature].to_s
end
@@insert_entry_ids.each do |tp|
path = Path.new(tp[:path])
.relative_to(Config.current.library_path).to_s
conn.exec "insert into ids (id, path, signature, " \
"unavailable) values (?, ?, ?, 0)",
tp[:id], path, tp[:signature].to_s
end
end
end
@insert_ids.clear
@@insert_entry_ids.clear
@@insert_title_ids.clear
end
end
@@ -336,7 +403,8 @@ class Storage
tags = [] of String
MainFiber.run do
get_db do |db|
db.query "select distinct tag from tags" do |rs|
db.query "select distinct tag from tags natural join titles " \
"where unavailable = 0" do |rs|
rs.each do
tags << rs.read String
end
@@ -368,44 +436,90 @@ class Storage
end
end
def optimize
def mark_unavailable
MainFiber.run do
Logger.info "Starting DB optimization"
get_db do |db|
# Detect dangling entry IDs
trash_ids = [] of String
db.query "select path, id from ids" do |rs|
db.query "select path, id from ids where unavailable = 0" do |rs|
rs.each do
path = rs.read String
trash_ids << rs.read String unless File.exists? path
fullpath = Path.new(path).expand(Config.current.library_path).to_s
trash_ids << rs.read String unless File.exists? fullpath
end
end
# Delete dangling IDs
db.exec "delete from ids where id in " \
unless trash_ids.empty?
Logger.debug "Marking #{trash_ids.size} entries as unavailable"
end
db.exec "update ids set unavailable = 1 where id in " \
"(#{trash_ids.map { |i| "'#{i}'" }.join ","})"
Logger.debug "#{trash_ids.size} dangling IDs deleted" \
if trash_ids.size > 0
# Delete dangling thumbnails
trash_thumbnails_count = db.query_one "select count(*) from " \
"thumbnails where id not in " \
"(select id from ids)", as: Int32
if trash_thumbnails_count > 0
db.exec "delete from thumbnails where id not in (select id from ids)"
Logger.info "#{trash_thumbnails_count} dangling thumbnails deleted"
# Detect dangling title IDs
trash_titles = [] of String
db.query "select path, id from titles where unavailable = 0" do |rs|
rs.each do
path = rs.read String
fullpath = Path.new(path).expand(Config.current.library_path).to_s
trash_titles << rs.read String unless Dir.exists? fullpath
end
end
# Delete dangling tags
trash_tags_count = db.query_one "select count(*) from tags " \
"where id not in " \
"(select id from ids)", as: Int32
if trash_tags_count > 0
db.exec "delete from tags where id not in (select id from ids)"
Logger.info "#{trash_tags_count} dangling tags deleted"
unless trash_titles.empty?
Logger.debug "Marking #{trash_titles.size} titles as unavailable"
end
db.exec "update titles set unavailable = 1 where id in " \
"(#{trash_titles.map { |i| "'#{i}'" }.join ","})"
end
end
end
private def get_missing(tablename)
ary = [] of IDTuple
MainFiber.run do
get_db do |db|
db.query "select id, path, signature from #{tablename} " \
"where unavailable = 1" do |rs|
rs.each do
ary << {
id: rs.read(String),
path: rs.read(String),
signature: rs.read(String?),
}
end
end
end
Logger.info "DB optimization finished"
end
ary
end
private def delete_missing(tablename, id : String? = nil)
MainFiber.run do
get_db do |db|
if id
db.exec "delete from #{tablename} where id = (?) " \
"and unavailable = 1", id
else
db.exec "delete from #{tablename} where unavailable = 1"
end
end
end
end
def missing_entries
get_missing "ids"
end
def missing_titles
get_missing "titles"
end
def delete_missing_entry(id = nil)
delete_missing "ids", id
end
def delete_missing_title(id = nil)
delete_missing "titles", id
end
def close

51
src/util/signature.cr Normal file
View File

@@ -0,0 +1,51 @@
require "./util"
class File
abstract struct Info
def inode : UInt64
@stat.st_ino.to_u64
end
end
# Returns the signature of the file at filename.
# When it is not a supported file, returns 0. Otherwise, uses the inode
# number as its signature. On most file systems, the inode number is
# preserved even when the file is renamed, moved or edited.
# Some cases that would cause the inode number to change:
# - Reboot/remount on some file systems
# - Replaced with a copied file
# - Moved to a different device
# Since we are also using the relative paths to match ids, we won't lose
# information as long as the above changes do not happen together with
# a file/folder rename, with no library scan in between.
def self.signature(filename) : UInt64
if is_supported_file filename
File.info(filename).inode
else
0u64
end
end
end
class Dir
# Returns the signature of the directory at dirname. See the comments for
# `File.signature` for more information.
def self.signature(dirname) : UInt64
signatures = [File.info(dirname).inode]
self.open dirname do |dir|
dir.entries.each do |fn|
next if fn.starts_with? "."
path = File.join dirname, fn
if File.directory? path
signatures << Dir.signature path
else
_sig = File.signature path
# Only add its signature value to `signatures` when it is a
# supported file
signatures << _sig if _sig > 0
end
end
end
Digest::CRC32.checksum(signatures.sort.join).to_u64
end
end

View File

@@ -2,6 +2,7 @@ IMGS_PER_PAGE = 5
ENTRIES_IN_HOME_SECTIONS = 8
UPLOAD_URL_PREFIX = "/uploads"
STATIC_DIRS = ["/css", "/js", "/img", "/favicon.ico"]
SUPPORTED_FILE_EXTNAMES = [".zip", ".cbz", ".rar", ".cbr"]
def random_str
UUID.random.to_s.gsub "-", ""
@@ -31,6 +32,10 @@ def register_mime_types
end
end
def is_supported_file(path)
SUPPORTED_FILE_EXTNAMES.includes? File.extname(path).downcase
end
struct Int
def or(other : Int)
if self == 0
@@ -92,3 +97,18 @@ def sort_titles(titles : Array(Title), opt : SortOptions, username : String)
ary
end
class String
# Returns the similarity (in [0, 1]) of two paths.
# For the two paths, separate them into arrays of components, count the
# number of matching components backwards, and divide the count by the
# number of components of the shorter path.
def components_similarity(other : String) : Float64
s, l = [self, other]
.map { |str| Path.new(str).parts }
.sort_by &.size
match = s.reverse.zip(l.reverse).count { |a, b| a == b }
match / s.size
end
end

View File

@@ -1,19 +1,24 @@
# Web related helper functions/macros
# This macro defines `is_admin` when used
macro check_admin_access
is_admin = false
# The token (if exists) takes precedence over the default user option.
# this is why we check the default username first before checking the
# token.
if Config.current.disable_login
is_admin = Storage.default.
username_is_admin Config.current.default_username
end
if token = env.session.string? "token"
is_admin = Storage.default.verify_admin token
end
end
macro layout(name)
base_url = Config.current.base_url
check_admin_access
begin
is_admin = false
# The token (if exists) takes precedence over the default user option.
# this is why we check the default username first before checking the
# token.
if Config.current.disable_login
is_admin = Storage.default.
username_is_admin Config.current.default_username
end
if token = env.session.string? "token"
is_admin = Storage.default.verify_admin token
end
page = {{name}}
render "src/views/#{{{name}}}.html.ecr", "src/views/layout.html.ecr"
rescue e
@@ -24,6 +29,15 @@ macro layout(name)
end
end
macro send_error_page(msg)
message = {{msg}}
base_url = Config.current.base_url
check_admin_access
page = "Error"
html = render "src/views/message.html.ecr", "src/views/layout.html.ecr"
send_file env, html.to_slice, "text/html"
end
macro send_img(env, img)
send_file {{env}}, {{img}}.data, {{img}}.mime
end

View File

@@ -1,5 +1,13 @@
<ul class="uk-list uk-list-large uk-list-divider" x-data="component()" x-init="init()">
<li><a class="uk-link-reset" href="<%= base_url %>admin/user">User Management</a></li>
<li>
<a class="uk-link-reset" href="<%= base_url %>admin/missing">Missing Items</a>
<% if missing_count > 0 %>
<div class="uk-align-right">
<span class="uk-badge"><%= missing_count %></span>
</div>
<% end %>
</li>
<li>
<a class="uk-link-reset" @click="scan()">
<span :style="`${scanning ? 'color:grey' : ''}`">Scan Library Files</span>
@@ -19,7 +27,7 @@
</li>
<li>
<span>Theme</span>
<select id="theme-select" class="uk-select uk-align-right uk-width-1-3@m uk-width-1-2" :val="themeSetting" @change="themeChanged($event)">
<select id="theme-select" class="uk-select uk-align-right uk-width-1-3@m uk-width-1-2" :value="themeSetting" @change="themeChanged($event)">
<option>Dark</option>
<option>Light</option>
<option>System</option>

View File

@@ -8,7 +8,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<redoc spec-url="/openapi.json"></redoc>
<redoc spec-url="<%= base_url %>openapi.json"></redoc>
<script src="https://cdn.jsdelivr.net/npm/redoc/bundles/redoc.standalone.js"></script>
</body>
</html>

View File

@@ -1,12 +0,0 @@
<div class="uk-margin" x-data="tagsComponent()" x-cloak x-init="load(<%= is_admin %>)">
<p class="uk-text-meta" @selectstart.prevent>
<span style="position:relative; bottom:3px; margin-right:5px;">Tags: </span>
<template x-for="tag in tags" :key="tag">
<span class="uk-label uk-label-primary" style="padding:2px 5px; margin:0 5px 5px 5px; text-transform:none;">
<a class="uk-link-reset" x-show="isAdmin" @click="rm($event)" :id="`${tag}-rm`"><span uk-icon="close" style="margin-right: 5px; position: relative; bottom: 1.5px;"></span></a><a class="uk-link-reset" x-text="tag" :href="`<%= base_url %>tags/${encodeURIComponent(tag)}`"></a>
</span>
</template>
<a class="uk-link-reset" style="position:relative; bottom:3px;" :uk-icon="inputShown ? 'close' : 'plus'" @click="toggleInput($nextTick)" x-show="isAdmin"></a>
</p>
<input id="tag-input" class="uk-input" type="text" placeholder="Type in a new tag and hit enter" x-model="newTag" @keydown="keydown($event)" x-show="inputShown">
</div>

View File

@@ -43,7 +43,7 @@
<template x-if="job.status_message.length > 0">
<div class="uk-inline">
<span uk-icon="info"></span>
<div uk-dropdown x-text="job.status_message"></div>
<div uk-dropdown x-text="job.status_message" style="white-space: pre-line;"></div>
</div>
</template>
</td>

View File

@@ -37,7 +37,7 @@
<div class="uk-navbar-toggle" uk-navbar-toggle-icon="uk-navbar-toggle-icon" uk-toggle="target: #mobile-nav"></div>
</div>
<div class="uk-navbar-left uk-visible@s">
<a class="uk-navbar-item uk-logo" href="<%= base_url %>"><img src="<%= base_url %>img/icon.png"></a>
<a class="uk-navbar-item uk-logo" href="<%= base_url %>"><img src="<%= base_url %>img/icon.png" style="width:90px;height:90px;"></a>
<ul class="uk-navbar-nav">
<li><a href="<%= base_url %>">Home</a></li>
<li><a href="<%= base_url %>library">Library</a></li>
@@ -69,7 +69,7 @@
</div>
<div class="uk-section uk-section-small">
</div>
<div class="uk-section uk-section-small" id="main-section">
<div class="uk-section uk-section-small" style="position:relative;">
<div class="uk-container uk-container-small">
<div id="alert"></div>
<%= content %>

View File

@@ -0,0 +1,40 @@
<div x-data="component()" x-init="load()" x-cloak x-show="!loading">
<p x-show="empty" class="uk-text-lead uk-text-center">No missing items found.</p>
<div x-show="!empty">
<p>The following items were present in your library, but now we can't find them anymore. If you deleted them mistakenly, try to recover the files or folders, put them back to where they were, and rescan the library. Otherwise, you can safely delete them and the associated metadata using the buttons below to free up database space.</p>
<button class="uk-button uk-button-danger" @click="rmAll()">Delete All</button>
<table class="uk-table uk-table-striped uk-overflow-auto">
<thead>
<tr>
<th>Type</th>
<th>Relative Path</th>
<th>ID</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
<template x-for="title in titles" :key="title">
<tr :id="`title-${title.id}`">
<td>Title</td>
<td x-text="title.path"></td>
<td x-text="title.id"></td>
<td><a @click="rm($event)" uk-icon="trash"></a></td>
</tr>
</template>
<template x-for="entry in entries" :key="entry">
<tr :id="`entry-${entry.id}`">
<td>Entry</td>
<td x-text="entry.path"></td>
<td x-text="entry.id"></td>
<td><a @click="rm($event)" uk-icon="trash"></a></td>
</tr>
</template>
</tbody>
</table>
</div>
</div>
<% content_for "script" do %>
<script src="<%= base_url %>js/alert.js"></script>
<script src="<%= base_url %>js/missing-items.js"></script>
<% end %>

View File

@@ -111,7 +111,8 @@
<style>
img[data-src][src*='data:image'] { background: white; }
#root img { width: 100%; }
img { width: 100%; }
.reader-bg { background: black; }
</style>
</html>

View File

@@ -34,7 +34,10 @@
</ul>
<p class="uk-text-meta"><%= title.content_label %> found</p>
<%= render_component "tags" %>
<div class="uk-margin" x-data="tagsComponent()" x-cloak x-init="load(<%= is_admin %>)" x-show="!loading">
<select class="tag-select" multiple="multiple" style="width:100%">
</select>
</div>
<div class="uk-grid-small" uk-grid>
<div class="uk-margin-bottom uk-width-3-4@s">
@@ -121,6 +124,9 @@
<% content_for "script" do %>
<%= render_component "dots-scripts" %>
<link href="https://cdn.jsdelivr.net/npm/select2@4.1.0-beta.1/dist/css/select2.min.css" rel="stylesheet" />
<link href="<%= base_url %>css/tags.css" rel="stylesheet" />
<script src="https://cdn.jsdelivr.net/npm/select2@4.1.0-beta.1/dist/js/select2.min.js"></script>
<script src="<%= base_url %>js/alert.js"></script>
<script src="<%= base_url %>js/title.js"></script>
<script src="<%= base_url %>js/search.js"></script>